Mario Richtsfeld, Wolfgang Ponweiser and Markus Vincze
Institute of Automation and Control, Vienna University of Technology
Gusshausstr. 27-29, Vienna, Austria
Service robotics, laser-range scanning, object detection, task planning.
In the near future, service robots will support people with different handicaps to improve the quality of their
life. One of the required key technologies is to setup the grasping ability of the robot. This includes an
autonomous object detection and grasp motion planning to fulfil the task of providing objects from any position
on a table to the user. This paper presents a complete system, which consists of a fixed working station
equipped with a laser-range scanner, a seven degrees of freedom arm manipulator and an arm prothesis as
gripper. The contribution of this work is to use only one sensor system based on a laser-range scanning head
to solve this challenge. The goal is that the user can select any defined object on the table and the robot arm
delivers it to a target position or to the disabled person.
At the beginning of the 1970’s the development
of service and rehabilitation robots started to sup-
port disabled people in their daily life. The
goal is to make them more independent. To-
day we differ between fixed systems, in which
an industrial robot is mounted on a working sta-
tion and mobile systems, e.g. wheelchair mounted
manipulators, like MANUS (Mokhtari, 2001) or
FRIEND-I (Martens, 2001) and FRIEND-II (Ivlev,
2005). Popular fixed systems are e.g. De-
Var (Van der Loos, 1995), ProVar (Van der Loos,
1999), RAID (Eftring, 1994), MASTER-RAID (Dall-
away, 1995) or CAPDI (Casals, 1999).
Our vision is a fully autonomous mobile robot,
which is able to detect, grasp and manipulate any kind
of object. One of the key challenges of this work is
the robust perception of objects. This challenge is
analyzed by a fixed setup consisting of a laser-range
scanner and a robot arm. We use an AMTEC
arm with seven degrees of freedom, which is used for
object grasping and manipulation. The joint setup is
assembled similar to a human arm. The robot arm
is equipped with a hand prothesis from the company
Otto Bock
, which we are using as gripper. It is
thought that elderly persons will accept this type of
gripper more easily than an industrial gripper, due to
the form and the optical characteristics.
The outline of the paper is as follows: In the next
section the state of the art of grasp robot systems,
grasping technology and object perception based on
structure in 2-d and 3-d is presented. Section 3 in-
troduces our robotic system and its components. Sec-
tion 4 describes the object identification to calculate
the object position and Section 5 details the grasping
and manipulation. Section 6 gives some experimental
results during a live demo presentation and Section 7
finally concludes the paper.
In the early 1970’s one of the first wheelchair
mounted manipulator was developed at the V.A. Re-
habilitation Engineering (formerly Prosthetics) cen-
ter (Prior, 1993). From 1983 to 1988 the mobile
manipulator MoVAR (Van der Loos, 1995) was de-
veloped. This PUMA-250 robot was instrumented
with a camera for remote sensing, a six-axis force
sensor and a gripper with finger pad-mounted prox-
imity sensors. A nice overview of different systems,
such as the Wolfson-Robot and the Wessex-Robot is
given by Hagen and Hillmann (Hagan, 1997). Up
Richtsfeld M., Ponweiser W. and Vincze M. (2008).
In Proceedings of the Fifth International Conference on Informatics in Control, Automation and Robotics - RA, pages 165-170
DOI: 10.5220/0001481101650170
to now a number of scientists have been working
on the same idea to develop a wheelchair mounted
robot or a mobile robot system with arms to han-
dle objects and assist elderly and handicapped per-
sons, e.g. (Martens, 2001), (Volosyak, 2005). In the
FRIEND systems (Martens, 2001), (Ivlev, 2005) the
robot arm is controlled by a PC, which is fixed on the
backside of the wheelchair. Both systems use a stereo
camera system for object detection. The user inter-
action is based on a LC-display. The object must be
placed on a predefined position on a tray, mounted at
the front side of the wheelchair. A successful execu-
tion of the grasping task in this system is only pos-
sible for similar types of objects. Additionally they
developed a ”smart tray” that is used in combination
with the vision sensors. This ”smart tray” measures
the weight and the position of objects with a matrix
foil position sensor.
In comparison to the FRIEND systems, Saxena et
al. (Saxena, 2006) developed a learning algorithm that
predicts the grasp position of novel objects as a func-
tion of 2-d images, without building an explicit 3-d
model of the object. This algorithm is trained via su-
pervised learning using synthetic images for the train-
ing set. The work focuses on the task of identifying
grasping positions without taking any complex ma-
nipulation tasks into account. A similar system de-
scribes Miller et al. (Miller, 2003). Their work speci-
fies an automatic grasp planning system for hand con-
figurations using shape primitives. By modeling an
object as a sphere, cylinder, cone or box. They also
use a set of rules to generate grasp positions.
In our case the vision task is to detect edges of
objects that indicate grasp points. Accurate 3-d data
is achieved by direct depth measurements, like laser-
range scanning. In the range images, grasp points are
indicated by object edges and grasp surface patches.
Wang et al. (Wang, 2005) developed a general frame-
work of automatic grasping of unknown objects by
incorporating a laser scanner and a simulation envi-
ronment. Their algorithms need a lot of time to de-
tect grasp points. To aid industrial bin picking tasks
Boughorbel et al. (Boughorbel, 2007) developed a
system that provides accurate 3-d models of parts and
objects in the bin to realize precise grasping opera-
tions. Due to their superquadrics based object mod-
elling approach only rotation-symmetric objects can
be used. To that effect Biegelbauer et al. describes
a new approach of a hierarchical RANSAC search
to obtain fast detection results of objects, which are
modeled using approximated Superquadrics (Biegel-
bauer, 2007).
One of the most fundamental techniques for edge
detection in range images is the scan line approxima-
tion (Jiang, 1999). It is well known and more ef-
ficient than the standard Canny (Canny, 1986) algo-
rithm. The raw data points are approximated by a set
of bivariate polynomial functions, in which the dis-
continuity of the fitted functions indicate the edge po-
sition. Katsoulas (Katsoulas, 2004) proposed an im-
proved scan line approach by using an additional sta-
tistical merging step for a better handling of outliers.
Based on these techniques we developed a 3-d edge
detection method that enables a faster cylinder fit in
3-d range data.
The goal is, that the user can select any object on a
table and the robot arm delivers it to a defined posi-
tion or to the disabled person. The main challenges
to solve are the robust detection of edges and their in-
terpretation as grasping points. Our approach is based
on scanning the objects by a rotating laser-range scan-
ner and execution of subsequent path planning and
grasping motion. Hence the system consists of a
pan/tilt-mounted red-light laser and scanning camera
and a seven degrees of freedom robot arm, which
is equipped with a human like prosthesis hand (see
Fig. 1).
Figure 1: Overview of the system components and their in-
3.1 Laser-Range Scanner
The laser-range scanner records a snap-shot of the ob-
ject scene with the help of a pan/tilt-unit. At present,
it is mounted on a table. We are working to miniatur-
ize the laser-range scanner to mount it on the shoul-
der of the robot later. A high resolution sensor is
needed in order to detect a reasonable number of
ICINCO 2008 - International Conference on Informatics in Control, Automation and Robotics
edge-points of the objects with the required accuracy.
The laser-range scanner used for this work consists
of a red-light LASIRIS laser from StockerYale
635nm and a MAPP2500 CCD-camera from SICK-
mounted on a pan/tilt-unit (PowerCube Wrist
from AMTEC robotics). With the help of a cylin-
der lens, the laser-light is expanded and moves hor-
izontally over the scene of interest. The camera grabs
the laser-light profiles and extracts the laser-lines with
the integrated microprocessor. The 3-d data is trans-
formed to the world coordinate system. Finally the
result can be displayed as a point cloud (see Fig. 2).
Figure 2: Exposure of the raw point cloud with 75.863
voxel. The two shadows from laser and camera are clearly
3.2 Robot Arm and Gripper
For this work we use the ”Light Weight Arm 7 DOF”
from AMTEC robotics and a hand prothesis from
Otto Bock as gripper. The robot arm exhibits seven
degrees of freedom with a joint configuration simi-
larly to the human arm (shoulder, elbow and wrist).
The seventh degree of freedom is required to enable
complex object grasping and manipulation and allow
for some flexibility to avoid obstacles. The prosthe-
sis as end effector is selected due to the integrated
force sensors as well as its increased acceptance of
elderly and handicapped persons. It has three active
fingers, the thumb, the index finger and the middle fin-
ger. The last two fingers are just for cosmetic reasons.
Since, they have no active function in the grasping
process their uncontrollable behavior must be consid-
ered, which reduces the grasping radius (see Fig. 1).
As a huge advantage the integrated tactile sensors are
used to detect a potential sliding of objects, which ini-
tializes a readjustment of the fingers.
3.3 Operation Sequence
The first step is to scan the scene on the table by the
laser-range scanner. The camera converts the laser-
profiles to a 3-d point cloud, which can be visualized.
Now the user can select the desired object. The de-
veloped algorithm analyzes the point cloud and calcu-
lates the position of the searched object. A commer-
cial path planning tool from AMROSE
calculates the
trajectory to grasp the object. Before the robot arm
delivers the object, the user can check the calculated
trajectory in a simulation sequence. Then the robot
arm executes the off-line programmed trajectory. The
algorithm is implemented in C++. For displaying the
results the Visualization Tool Kit (VTK)
is used.
The main goal of our work is to robustly detect cylin-
drical objects in the recorded point cloud in real
time. Robustness includes the positive detection of
defined objects despite any noise and outliers in a
point cloud, which can be caused by specular surfaces
(see Fig. 2, edges of the objects). To reduce complex-
ity we only consider cylindrical objects for object de-
tection for this work. An additional challenge is the
complex interaction between the different operation
parts. Finally to keep the standby time acceptable for
the user the complete operating cycle should be fin-
ished within 20sec.. This time limit is challenging
since usual object detection starts with an exhaustive
segmentation step. As an example, object segmen-
tation alone by recursive flood-filling with region-
octree (Burger, 2007) of the desired table scene takes
more than 30sec. (see Fig. 3). Thus a faster solution
must be found. One alternative, which we exploited
in our work is based on well investigated curvatures.
Fig. 4 presents the steps of the fast object detection
Figure 3: Segmentation of the different objects by recursive
flood-filling. Images are best viewed in color.
method. In the first step, the ”raw data preprocessing
and vector estimation” the raw data points are pre-
processed with a low pass filter to reduce any noise.
Freely available open source software,
One of the most time consuming calculations is the
normal vector estimation based on the orientation of
the local neighborhood of 20mm, for what a region-
octree is used. These vectors are required to compute
the axis of the cylinder objects. A lot of tests have
shown that for a neighborhood of 20mm a reason-
able accuracy can be achieved, while the calculation
time stays acceptable. The ”range image segmenta-
Figure 4: Flow chart of the object detection approach.
tion” starts by detecting the surface of the table with
a RANSAC (Fischler, 1981) based plane fit. Then we
analyze the curvature of the remaining points to filter
neighbouring voxels with an angle difference between
and ±90
(see Fig. 5) .
The ”fast cylinder fit” starts with a RANSAC based
circle fit. Randomly three high curvature points are
picked. The resulting circle is extended to a poten-
tial cylinder along its circumscribed axis down to the
table. For every vicinity point, within a defined dis-
tance of 2mm of the calculated cylinder barrel, the
normal distance to the cylinder barrel is calculated.
The trial with the lowest mean of these distances is se-
lected as cylinder (see Fig. 6). For comparison Jiang
et al. (Jiang, 2005) published a method for 3-d circle
Figure 5: The acquired range image of the current table
scene. The points with a high curvature are marked with
blue dots. Images are best viewed in color.
Figure 6: Detected objects in the table scene (blue cylinder
- spray-on glue, red cylinder - beverage can, green points -
rigid obstacles). Images are best viewed in color.
fitting. They reduce the number of local minima, but
the error function is no more Euclidian. Here another
simple proposal with an Euclidian function is used.
For an explicit description, the raw data points of a
profile scan are defined as (xi, yi, zi), n is the number
of voxels and (xa, ya, za) is the circle’s center. The
resulting error function e is:
e =
(xi xa)
+ (yi ya)
+ (zi za)
) r
The error must be smaller or equal than a defined
threshold. In our case we use a distance of 2mm:
|e| 2mm (2)
In the last step of Fig. 4 ”Transmission of the Calcu-
lated Object Position to the Path Planning Tool” the
calculated object position in the actual environment
model for collision avoidance has to be transmitted to
the path planning tool. This 3-d mesh is generated
by using all objects besides the target object, based
on the triangles calculated by a DeLaunay triangula-
tion (O’Rourke, 1998) This step is important to enable
a collision free robot trajectory.
The task of this part of our work is to calculate a col-
lision free robot path and to execute the grasping ac-
tivity safely. The first step is performed by the path
planning tool from AMROSE. The input to this tool is
the detected object pose, the environment model and
a transformation between the robot coordination sys-
tem and the range scanner coordinate system. The
output is a collision free trajectory to the desired ob-
ject. Before the robot execution is approved, the user
can check a simulation of the calculated trajectory and
decide, if it is safe enough to handle the object (see
Fig. 7 and Fig. 8). After the robot approaches the user
ICINCO 2008 - International Conference on Informatics in Control, Automation and Robotics
Figure 7: Visualization of the trajectory by a simulation
tool. The white cylinder is the grasping object. The green
cylinder (= 2.
grasping object) and the blue objects are
the obstacles. Images are best viewed in color.
Figure 8: Real position of the robot arm after the approach
can initiate the closing of the gripper. As soon as the
gripper encloses the object, the robot motion to the
transfer point starts. Finally the desired object can be
placed at a defined position or directly handed over to
the user.
The calculation of the object detection and local-
ization is performed by a PC with 1.8GHz Pentium
IV processor and takes less than 12sec. depending on
the range image size. The reliability depends on the
ambient light, object surface properties, laser beam
reflections and vibrations. Therefore, the laser-range
scanner must be configured to the respective environ-
ment. By using an additional red-light filter the im-
pact of light or reflections can be minimized.
The entire system exhibits its robust behavior and has
been evaluated at a live demo presentation
in front of
more than 1000 college students. During the demon-
stration day about 50 runs were performed. The main
problem that rarely appeared was a malfunction of
the path planning tool, because no suitable trajectory
could be found. Whereby the path planning had to be
restarted. Sometimes the last two fingers, which re-
duce the grasping radius (see Fig. 1), shift the grasp
object, but without a final effect on the success of the
grasping process. Tab. 1 shows a short analysis of the
arisen problems within 50 runs. The recoginition of
the cylindrical objects fails at strong environmental
influences by the ambient light.
The autonomous grasping function should be able
to find and grasp a cylindrical object in a defined area.
When objects are positioned closer to each other, the
autonomous grasping function show up difficulties to
find the correct object. A minimum distance of 20mm
(this distance is equal to the diameter of the thumb
of the hand prothesis) has to be observed between the
Table 1: Evaluation of the arisen problems in percent [%] at
50 runs.
Arison Problems Number of Events Percent [%]
Path Planning 11 22%
Hand Prosthesis 4 8%
Object Recognition 2 4%
Sum 17 34%
This paper presents an approach of a robot system
equipped with a laser-range scanner to get high accu-
racy table scene sensing. It shows that feature detec-
tion, in our case we only consider cylindrical objects,
is a faster way (12sec.), than usual object segmenta-
tion (more than 30sec.) by a flood-filling recursive
function. The presented method performs with very
high reliability. Thus the approach for object detec-
tion and localization is well suited for use in related
applications under difficult conditions.
A seven degrees of freedom arm manipulator and
an arm prothesis as gripper are used to grasp and de-
liver the desired object. The goal of this system is
to analyze the feasibility and reliability of object de-
tection, which could be shown at a live demo. The
cylinder detection approach can be extended to detect
any type of object, since it is based on a grouping of
high curvature points. This grasping approach can be
applied for any kind of geometrical figures. This will
expand the application to other tasks.
In the future, the robot arm will be installed on a
mobile robot and for the object detection we calculate
the grasping points of novel objects. This includes
the revision of the path planning tool and a segmenta-
tion of sharp curvature points to speed up the method.
Summarizing, this work illustrates that the concept of
a 3-d vision guided robot arm can be adopted to many
applications and has high potential to enable a more
complex system. We will also deal with the devel-
opment and the prototypes integration of a new laser
range sensor with additional two cameras for stereo-
vision to increase the robustness and predictability of
the object detection system.
Biegelbauer, G.; Vincze, M. (2007). Efficient 3d object de-
tection by fitting superquadrics to range image data
for robot’s object manipulation. In International Con-
ference on Robotics and Automation / ICRA, pages
1086–1091. IEEE Press.
Boughorbel, F.; Zhang, Y. (2007). Laser ranging and video
imaging for bin picking. In Assembly Automation, vol-
ume 23, pages 53–59.
Burger, W.; Burge, M. (2007). Digital Image Processing
- An Algorithmic Introduction Using Java. Springer,
UK, London, 1st edition.
Canny, J. (1986). A computational approach to edge detec-
tion. In Transactions on Pattern Analysis and Machine
Intelligence, volume 8, pages 679–698. IEEE Press.
Casals, A.; Merchan, R. (1999). Capdi: A robotized kitchen
for the disabled and elderly people. In Proceedings
of the 5th European Conference for the Advancement
Assistive Technology / AAATE, pages 346–351.
Dallaway, J.L.; Jackson, R. (1995). Rehabilitation robotics
in europe. In Transactions on Rehabilitation Engi-
neering, pages 33–45. IEEE Press.
Eftring, H. (1994). Robot control methods and results from
user trials on the raid workstation. In Proceedings
of 4th Int. Conf. on Rehabilitation Robotics / ICORR,
pages 97–101.
Fischler, M.A.; Boles, R. (1981). Random samples and con-
sensus: A paradigm for model fitting with applications
to image analysis and automated cartography. In Com-
munications of the ACM, volume 24, pages 381–395.
Hagan, K.; Hillman, M. (1997). The design of a wheelchair
mounted robot. In Colloquium on Computers in the
Service of Mankind: Helping the Disabled, pages 1–
Ivlev, O.; Martens, C. (2005). Rehabilitation robots friend-i
and friend-i with the dexterous lightweight manipula-
tor. volume 17, pages 111–123. IOS Press.
Jiang, X.; Cheng, D.-C. (2005). Fitting of 3d circles and el-
lipses using a parameter decomposition approach. In
5th International Conference on 3cartography Imag-
ing and Modeling / 3DIM, pages 103–109. IEEE
Jiang, X.; Bunke, H. (1999). Edge detection in range im-
ages based on scan line approximation. In Computer
Vision and Image Understanding, volume 73, pages
Katsoulas, D.; Werber, A. (2004). Edge detection in range
images of piled box-like objects. In Conference on
Pattern Recognition, volume 2, pages 80–84.
Martens, C.; Ruchel, N. (2001). A friend for assisting hand-
icapped people. In Robotics and Automation Maga-
zine, volume 8, pages 57–65. IEEE Press.
Miller, A.T.; Knoop, S. (2003). Automatic grasp planning
using shape primitives. In International Conference
on Robotics and Automation / ICRA, volume 2, pages
1824–1829. IEEE Press.
Mokhtari, M.; Abdurazak, B. (2001). Assistive technol-
ogy for the disabled people: Should it work? the
french approach. In International Journal of Assistive
Robotics and Mechatronics, volume 2, pages 26–32.
O’Rourke, J. (1998). Computational geometry in C. Univ.
Press, Cambridge, 2nd edition.
Prior, S.D.; Warner, P. (1993). Wheelchair-mounted robots
for the home environment. In Conference on Intel-
ligent Robots and Systems / IROS, volume 2, pages
1194–1200. IEEE Press.
Saxena, A.; Driemeyer, J. (2006). Learning to grasp novel
objects using vision. In RSS Workshop on Manipula-
tion for Human Environments.
Van der Loos, H. (1995). Va/stanford rehabilitation robotics
research and development program: Lessons learned
in the application of robotics technology to field of
rehabilitation. In Transactions on Rehabilitation En-
gineering, volume 3, pages 46–55. IEEE Press.
Van der Loos, H.; Wagner, J. (1999). Provar assistive robot
system architecture. In International Conference on
Robotics and Automation / ICRA, volume 1, pages
741–746. IEEE Press.
Volosyak, I.; Ivlev, O. (2005). Rehabilitation robot friend-
ii - the general concept and current implementation.
In 9th International Conference on Rehabilitation
Robotics / ICORR, pages 540–544.
Wang, B.; Jiang, L. (2005). Grasping unknown objects
based on 3d model reconstruction. In Proceedings
of International Conference on Advanced Intelligent
Mechatronics / ASME, pages 461–466. IEEE Press.
ICINCO 2008 - International Conference on Informatics in Control, Automation and Robotics