Smart Autonomous Part Displacement System based on Point Cloud

Segmentation

Eber Lawrence Souza Gouveia

, Rupal Srivastava

, Maulshree Singh

, Sean Lyons

Eddie Armstrong

and Declan Devine

Materials Research Institute, Technological University of the Shannon: Midlands Midwest, Athlone, Ireland

Johnson & Johnson, Advanced Technology Centre, University of Limerick, Limerick, Ireland

Keywords: Manufacturing Line, Smart System, Pick and Place Task, Computational Vision, Robot Operating System.

Abstract: Robotic arms are widely used in manufacturing lines to automate the manipulation of products, providing

many advantages, such as increasing production and minimizing labour costs. However, most robotic arms

operate in a controlled environment, executing predefined movements. Such a feature prevents the robot arm

from working in an environment where multiple product types are in different placements. In this way, this

concept paper describes the development of a smart robotic system capable of performing an autonomous

pick-and-place task of injected moulded parts from the first conveyor belt to the next, based on its spatial data

obtained from a 3D scanner. After obtaining the digital point cloud from the moulded part, the PointNet deep

learning model was used to segment and then extract the spatial position of its sprue, which is one of the

common structures of any moulded part. Finally, the robotic arm combined with its end-effector can pick up

these parts regardless of their shape, orientation, and size. The system proposed is composed of three

components, i.e., the IRB 1200 robotic arm from ABB, the PhoXi 3D Scanner from Photoneo, and the two-

finger gripper PB-0013 from Gimatic. Moreover, all system components were interconnected using Robot

Operating System as middleware. This concept paper discusses the setup and plan for the same.

1 INTRODUCTION

In recent years Industry 4.0 concept has become

popular, and many industries are changing their

factory process to adapt to this new concept. A few

years ago, terms such as Artificial Intelligence (AI),

robotics, cloud, Internet of Things (IoT), smart

factory were unknown to a large part of society.

However, due to the advances in technology, these

terms have become part of our activities of daily

living (Lasi et al., 2014; Oztemel & Gursev, 2020).

Such a term is also known as the fourth industrial

revolution, considering that it was a breakthrough in

industrial manufacturing (Lasi et al., 2014).

Industry 4.0 raises new meaningful concepts to

the industry process, making it more automated,

https://orcid.org/0000-0003-3766-2043

https://orcid.org/0000-0002-3127-4982

https://orcid.org/0000-0003-4788-1231

https://orcid.org/0000-0003-1998-070X

https://orcid.org/0000-0001-9396-210X

https://orcid.org/0000-0002-1364-5583

intelligent, and interconnected. Such features are

possible due to terms mentioned before, such as IoT,

smart factory, and cloud manufacturing. These

concepts enable various parts of a production line to

be interconnected controlled virtually (Ghobakhloo,

2020; Roblek et al., 2016). Moreover, industry 4.0

seeks to achieve new advantages compared to the

previous concepts, e.g., creation of new business

models, integrated and real-time operations, cost

reduction, energy savings, optimization of natural

resources, and reduction of errors (Bai et al., 2020;

Maskuriy et al., 2019; Oláh et al., 2020). In this way,

industry 4.0 brings opportunities to the current

business model for large and small companies.

Industrial robots are other elements that play a

fundamental role in Industry 4.0. More specifically,

Gouveia, E., Srivastava, R., Singh, M., Lyons, S., Armstrong, E. and Devine, D.

Smart Autonomous Part Displacement System based on Point Cloud Segmentation.

DOI: 10.5220/0011353100003271

In Proceedings of the 19th International Conference on Informatics in Control, Automation and Robotics (ICINCO 2022), pages 549-554

ISBN: 978-989-758-585-2; ISSN: 2184-2809

549

Figure 1: Applications of PointNet. Source: (Qi et al., 2016).

when it comes to pick-and-place tasks in

manufacturing lines, our focus is on robotic arms.

Robotic arms are widely used in factories to automate

the manipulation of products, providing many

advantages, such as increasing production and

minimizing labour costs (Borkar, 2017; Li et al.,

2018; Prabhakar et al., 2021; Ramanathan S et al.,

2020). Although they have been present in

manufacturing lines since the third industrial

revolution, robotic arms are even more robust in

Industry 4.0, becoming more intelligent, productive,

flexible, versatile, safer, and collaborative (Bahrin et

al., 2016). Therefore, robotic arms are becoming a

key component in the operation of smart factories as

they can complete cooperative tasks intelligently

(Ruchiand Goel & Pooja Gupta, 2020).

Pick and place robots have been present in

factories for decades due to their precision, high

speed, and cost‐effectiveness in repetitive tasks

compared to manual workers (Chettibi et al., 2004;

Perumaal & Jawahar, 2013). When incorporated with

Industry 4.0 features, these robots bring many

advantages for managing product manufacturing

lines, e.g., creating autonomous production lines.

However, most robotic arms present in the current

factories operate in a controlled environment,

executing predefined, repetitive movements and are

frequently referred to as ‘pick and place’ robots due

to their limited functionality. Such a feature prevents

the robot arm from working in conditions where

exists multiple product types in different placements.

Hence, this presented factory limitation provides the

opportunity for developing more intelligent

manufacturing line control systems.

In this article, we present the concept of a novel

smart robotic system capable of performing an

autonomous identification and displacement of

injected moulded parts based on their point cloud

obtained from a 3D scanner. After getting the point

cloud from the moulded part, the PointNet deep

learning model is used to extract the coordinates of

the part sprue, which is the general structure of any

moulded part. Finally, the robotic arm combined with

its end-effector must pick up this part regardless of its

shape, orientation, and size. The proposed system is

part of a manufacturing line, and it is composed of

three main components, i.e., the IRB 1200 robotic

arm from ABB (ABB, n.d.), the PhoXi 3D Scanner

from Photoneo (Photoneo, n.d.), and the PB-0013

pneumatic two-finger gripper from Gimatic (REF).

Moreover, all system components were

interconnected using Robot Operating System (ROS)

as middleware (Willow Garage et al., n.d.).

2 POINTNET NEURAL

NETWORK

PointNet is a neural network approach that deals with

point clouds. Unlike other methods that require image

grids or 3D voxels, the PointNet neural network

directly consumes point clouds, i.e., turning the

process highly efficient and effective (Qi et al., 2016).

Figure 1 shows the three possible applications of the

PointNet: classification, part segmentation and

semantic segmentation.

Point clouds have many applications, such as the

representation of the physical environment inside a

virtual one through the data obtained from sensors.

ICINCO 2022 - 19th International Conference on Informatics in Control, Automation and Robotics

550

Such application is widely seen in autonomous

systems that constantly need information about the

physical environment.

With the substantial increase in autonomous

systems, it becomes crucial to understand how to

work with point clouds. Therefore, PointNet neural

network has been a breakthrough in computer vision

due to its broad number of applications in many areas,

such as robotics and autonomous systems.

3 AUTONOMOUS PART

IDENTIFICATION AND

DISPLACEMENT SYSTEM

SETUP

As conveyed before, the autonomous part

displacement step of the manufacturing line is

composed of three components which are the robotic

arm, the pneumatic gripper, and the 3D scanner.

These components and their key features are

described in the sections below. Furthermore, the

IRC5 controller is another fundamental part of this

system once it controls external sensors and devices

through its I/O ports. Thus, it is also described with

more details in the next section.

Figure 2: Pick-and-place step components assembled in the

manufacturing line. (A) ABB 1200 robotic arm, (B) PhoXi

3D scanner, and (C) PB-0013 pneumatic two-finger

gripper.

3.1 IRB1200 Robotic Arm (A)

The IRB 1200 - 5/90 is the robotic arm used in the

presented manufacturing line. This robot has a reach

of 700 mm and can carry up to 5 kg of payload. Such

resources make this robotic arm ideal for a wide range

of industrial applications, including the proposed pick

and place task of moulded parts described in this

article.

3.2 PhoXi 3D Scanner (B)

The PhoXi S 3D Scanner from Photoneo is the

component designated for adding computer vision to

the manufacturing line, i.e., getting the point cloud of

the moulded parts. Such a scanner is ideal for high-

resolution, high-precision scanning of static scenes.

Its structured light projection technique provides

output in the form of point clouds for quick location

of any aimed object part.

3.3 PB-0013 Pneumatic Gripper (C)

The Gimatic PB-0013 is the End-Effector which is

attached to the robotic arm. Since it is a pneumatic

gripper, a compressor combined with a 3/2-way valve

is responsible for its actuation. Moreover, this gripper

is based on spring return, being normally open with a

closing grip torque of 80 N.cm at 6 bar pressure for

each jaw.

3.4 IRC5 Compact Controller

The IRC5 compact controller is the core component

of the presented system since it controls the I/O ports

and connects the robotic arm to ROS scripts. The

IRC5 compact controller uses the ABB’s RAPID

robot programming language. Despite being a high-

level programming language, RAPID has some

notable features, such as the possibility of connecting

it to other languages via socket.

4 METHODOLOGY

4.1 Hardware Overview

Figure 3 shows the hardware diagram representing

how the system components are interconnected. The

core component is the IRC5 compact controller,

which its function is to interconnect the system

components via I/O ports. Moreover, the controller is

connected to the laptop using a Local Area Network

(LAN) port, allowing its connection to ROS scripts

via socket.

The compressor, the 3/2-way valve, the robotic

arm and the end-effector compose the pneumatic side

of the system. First, a pneumatic hose connects the

compressor to the valve. Next, a second hose

connects the valve to a pneumatic input port of the

robotic arm. This port allows the airflow inside the

Smart Autonomous Part Displacement System based on Point Cloud Segmentation

551

Figure 3: System hardware overview.

robot, which has its pneumatic output port near

the robot tip. Finally, the last hose connects this

output port to the End-effector, closing the pneumatic

circuit. Moreover, a twenty-four volts output port

connects the pneumatic valve to the IRC5 compact

controller for switching whether the pressure is

released or suppressed.

The conveyor belt has a constant velocity and

uses another twenty-four volts output port of the

IRC5 compact controller to control whether this

conveyor belt is turned on or off. Moreover, an

ultrasonic sensor detects moulded parts going through

the conveyor belt, i.e., this sensor sends a high-level

signal to the controller via its input port every time a

moulded part goes through the conveyor belt. On the

other hand, the 3d scanner is the last part of the

hardware structure and is one of the most

fundamental components. The LAN port connects

this scanner to the laptop, allowing the execution of

its commands via ROS scripts, such as starting a new

scan or managing previous scans.

4.2 System Control Description

This system is being developed for performing on

manufacturing lines of moulded parts comes out from

injection machines. In this way, it is considered that

all objects placed in the conveyor belt have the

structure of a moulded part, i.e., sprue, runners, gates,

and products. Figure 4 shows two examples of

moulded parts and their structure parts.

Figure 5 shows the system flowchart,

representing the main steps of the autonomous part

displacement task. At the initial stage, the conveyor

belt starts and remains on until the ultrasonic sensor

detects a moulded part on this conveyor. Next, the

conveyor belt stops placing the moulded part below

the 3D scanner, getting the object point cloud through

its scan. Then, the scanner sends this point cloud to

the ROS scripts, starting the displacement stage of the

process.

Figure 4: Examples of injected moulded parts, showing

their standard components, i.e., sprue, runner, gate and

products.

The processing stage consists of the extraction of

information from the scanned object. More

specifically, the extraction of the sprue coordinates of

this moulded part. The first step of this processing

consists of segmenting the point cloud into the sprue,

ICINCO 2022 - 19th International Conference on Informatics in Control, Automation and Robotics

552

Figure 5: Autonomous pick-and-place process flowchart.

runners, gates, and products. In this way, the system

uses the PointNet algorithm for realizing this

segmentation since it is state-of-the-art in object

segmentation, fitting the aims of this work.

The PointNet model training is one of the biggest

challenges when dealing with this approach.

However, it is intended to create a dataset using the

ATOS Core 200 3D scanner, getting a precise and

reliable scan of a set of moulded parts. Finally, the

PointNet model will be trained and evaluated with

point clouds of moulded parts in real-time.

After the system starts to segment and then gets

the sprue coordinates, the End-effector attached to the

robotic arm must pick the moulded part by its sprue,

transferring it to the next conveyor belt or other

auxiliary process steps. This stage uses ROS-

Industrial and MoveIt [20] packages from ROS to

perform the robotic arm navigation. Using ROS-

Industrial is possible to control the robotic arm

sending the target coordinates. Moreover, MoveIt can

manage the navigation process, finding the best route

and avoiding collision points with the environment.

Finally, all the core steps occurring in the physical

layer are also represented digitally inside the Rviz

platform, allowing the visualization of these steps in

real-time using a computer.

5 DISCUSSION

The development of the proposed system is still

ongoing, and preliminary results will be presented

soon. Meanwhile, advances have been made in the

hardware assembly, interconnecting different system

parts, such as controlling the pneumatic valve and the

conveyor belt via the I/O ports of the IRC5 compact

controller. Furthermore, a digital representation of the

robotic arm, the 3D scanner and the conveyor belt

were created and then organized inside Rviz,

mimicking the physical manufacturing line.

Although the proposed system is being developed

for working in a manufacturing line of moulded parts,

it is vital to highlight that such an approach might be

spread for other scenarios using different objects

instead injected moulded parts. Owing to the

combination of the PointNet algorithm with 3D

vision, it is possible to highly increase the autonomy

of industrial processes, such as the gripping of objects

placed in a conveyor belt.

6 CONCLUSIONS

The current manuscript presents a conceptual

framework for an intelligent system for autonomous

part identification and displacement capable of self-

adjusting itself according to the injected moulded part

displacement on the conveyor belt. This feature

ensures the development of a more robust system that

is highly sensitive to the object's variations in its

shape and dimensions while working autonomously

with no or minimum human involvement. Moreover,

PointNet networks are state-of-the-art when dealing

with point cloud classification and segmentation,

making this network a suitable tool for pick-and-place

tasks in manufacturing lines.

ACKNOWLEDGEMENTS

This publication emanated from the research

conducted with the support of the Science Foundation

Ireland (SFI), Grant Number SFI 16/RC/3919, co-

funded by the European Regional Development Fund,

The Technological University of the Shannon

Presidents Doctoral Scholarship and Johnson &

Johnson.

REFERENCES

ABB. (n.d.). IRB-1200-5-90. Retrieved January 2, 2022,

from https://new.abb.com/products/robotics/industrial-

robots/irb-1200

Bahrin, M. A. K., Othman, M. F., Azli, N. H. N., & Talib,

M. F. (2016). Industry 4.0: A review on industrial

automation and robotic. In Jurnal Teknologi (Vol. 78,

Issues 6–13). https://doi.org/10.11113/jt.v78.9285

Bai, C., Dallasega, P., Orzes, G., & Sarkis, J. (2020).

Industry 4.0 technologies assessment: A sustainability

perspective. International Journal of Production

Smart Autonomous Part Displacement System based on Point Cloud Segmentation

553

Economics, 229. https://doi.org/10.1016/j.ijpe.2020.10

7776

Borkar, V. (2017). Develpoment of Pick and Place Robot

for Industrial. International Research Journal of

Engineering and Technology (IRJET), 4(9).

Chettibi, T., Lehtihet, H. E., Haddad, M., & Hanchi, S.

(2004). Minimum cost trajectory planning for industrial

robots. European Journal of Mechanics, A/Solids,

23(4).

https://doi.org/10.1016/j.euromechsol.2004.02.006

Ghobakhloo, M. (2020). Industry 4.0, digitization, and

opportunities for sustainability. In Journal of Cleaner

Production (Vol. 252). https://doi.org/10.1016/

j.jclepro.2019.119869

Lasi, H., Fettke, P., Kemper, H. G., Feld, T., & Hoffmann,

M. (2014). Industry 4.0. Business and Information

Systems Engineering, 6(4). https://doi.org/10.1007/

s12599-014-0334-4

Li, Y., Huang, T., & Chetwynd, D. G. (2018). An approach

for smooth trajectory planning of high-speed pick-and-

place parallel robots using quintic B-splines.

Mechanism and Machine Theory, 126. https://doi.org/

10.1016/j.mechmachtheory.2018.04.026

Maskuriy, R., Selamat, A., Ali, K. N., Maresova, P., &

Krejcar, O. (2019). Industry 4.0 for the construction

industry-How ready is the industry? Applied Sciences

(Switzerland), 9(14). https://doi.org/10.3390/app9142

819

Oláh, J., Aburumman, N., Popp, J., Khan, M. A., Haddad,

H., & Kitukutha, N. (2020). Impact of industry 4.0 on

environmental sustainability. Sustainability

(Switzerland), 12(11). https://doi.org/10.3390/su1211

4674

Oztemel, E., & Gursev, S. (2020). Literature review of

Industry 4.0 and related technologies. In Journal of

Intelligent Manufacturing (Vol. 31, Issue 1).

https://doi.org/10.1007/s10845-018-1433-8

Perumaal, S. S., & Jawahar, N. (2013). Automated

trajectory planner of industrial robot for pick-and-place

task. International Journal of Advanced Robotic

Systems, 10. https://doi.org/10.5772/53940

Photoneo. (n.d.). PhoXi 3D Scanner S. Retrieved January 2,

2022, from https://www.photoneo.com/products/

phoxi-scan-s/

Prabhakar, M., Paulraj, V., Karthi Kannappan, D. A.,

Dhanraj, J. A., & Ganapathy, D. (2021). Remote

Controlled Pick and Place Robot. IOP Conference

Series: Materials Science and Engineering, 1012(1).

https://doi.org/10.1088/1757-899x/1012/1/012003

Qi, C. R., Su, H., Mo, K., & Guibas, L. J. (2016). PointNet:

Deep Learning on Point Sets for 3D Classification and

Segmentation. CoRR, abs/1612.00593. http://arxiv.org/

abs/1612.00593

Ramanathan S, Akash V, Arun Prasad V, & Venkat Rahul

M. (2020). Pick and Place Robot for Surface Mounting

Devices. International Research Journal on Advanced

Science Hub, 2(8). https://doi.org/10.47392/irjash.20

20.97

Roblek, V., Meško, M., & Krapež, A. (2016). A Complex

View of Industry 4.0. SAGE Open, 6(2).

https://doi.org/10.1177/2158244016653987

Ruchiand Goel, & Pooja Gupta. (2020). Robotics and

Industry 4.0. In A. Nayyar Anand and Kumar (Ed.), A

Roadmap to Industry 4.0: Smart Production, Sharp

Business and Sustainable Development (pp. 157–169).

Springer International Publishing. https://doi.org/10.10

07/978-3-030-14544-6_9

Willow Garage, Stanford Artificial Intelligence Laboratory,

& Open Robotics. (n.d.). ROS - Robot Operating

System. Retrieved January 2, 2022, from

https://www.ros.org/

ICINCO 2022 - 19th International Conference on Informatics in Control, Automation and Robotics

554