Deﬁnition and Classiﬁcation of Primitives for the Robotic Unfolding of a

Piece of Clothing

Dimitra Triantafyllou and Nikos A. Aspragathos

Department of Mechanical Engineering and Aeronautics, University of Patras, Rio, Greece

Keywords:

Clothes, Unfolding, Primitives.

Abstract:

In this paper a dictionary of primitives is proposed that deﬁne critical features for the representation of a cloth’s

conﬁguration so as to unfold it using a general manipulator. The primitives are deﬁned, detected and classiﬁed

according to geometrical criteria. Experimental results demonstrate the effectiveness of the detection and

classiﬁcation approach. Finally, applications are presented where the dictionary of the proposed primitives

can be used, such as the deﬁnition of different types of folds of an unfolded fabric that can facilitate the

unfolding task. Conclusion and suggestions for future work completes this paper.

1 INTRODUCTION

Recent years robotic scientists tend to give augment-

ing attention to the ﬁeld of service robots. Robots

that engage house works could relief people from a

burden in their everyday life. A challenging area of

house works for robotic scientists is the manipulation

of non-rigid objects, such as clothes. Modelling and

predicting the behaviour of fabrics presents huge dif-

ﬁculties due to the large deformations and the very

low bending resistance of the fabrics.

A challenging task for the robotic handling of gar-

ments is their unfolding from a random conﬁguration

to a fully unfolded ﬂat state. Since there are inﬁ-

nite possible conﬁgurations in which a garment can be

found and there is no stable posture for grasping, as

in rigid objects, researchers propose ways to diminish

the garments’ possible conﬁgurations. The transition

of the garments into a group of known conﬁgurations

facilitates the recognition of their state, hence their

manipulation.

In order to achieve the reduction of the conﬁgura-

tions, Osawa et al. (Osawa et al., 2007) proposed a

technique, where the robot grasps a piece of clothing

from a random point, picks it up, detects its lowest

hanging corner, grasps it and then detects and grasps

its new lowest hanging corner. In this way, garments

result half folded in known conﬁgurations, facilitat-

ing their classiﬁcation to a type of clothing and their

unfolding. Cusumano-Towner et al. (Cusumano-

Towner et al., 2011) suggested re-grasping the cloth’s

lowest hanging point to reduce the possible conﬁgu-

rations. The goal was to recognize the speciﬁc article,

by matching its outline with existing templates, and

plan the sequence of manipulations, which bring the

article to the desired unfolded state.

On the other hand, Kaneko et al. (Kaneko and

Kakikura, 2001) used the relationships between the

shadows that appear on clothes and the convex shape

at the outline of clothing to detect points of the hem-

lines. Furthermore, they proposed a classiﬁcation

process, which matches the half folded clothes that

occur after grasping the garment from two hemline

points to predeﬁned templates that suggest the type

of clothing and its conﬁguration. Maintin-Shepard et

al. (Maitin-Shepard et al., 2010) implemented a vi-

sion algorithm that takes into account the sharpness

of curvature of the cloth in order to ﬁnd two neigh-

bour corners and manage the unfolding of towels.

Besides the unfolding task, some researchers de-

tect characteristic features met on clothes, such as

wrinkles or loose folds (Han and Zhu, 2005),(Ya-

mazaki and Inaba, 2009), (Yamazaki et al., 2011)

. Han et al (Han and Zhu, 2005) suggested a two

level generative model for cloth representation us-

ing shape from shading technique. An interesting as-

pect in their work is that they propose a dictionary of

shading and fold primitives in order to detect loose

folds on drapery and clothes. In previous work (Tri-

antafyllou and Aspragathos, 2011) we tried to deﬁne

and classify critical features on a piece of fabric. In

this paper, a better classiﬁcation with fewer categories

is introduced, simplifying the problem, and an efﬁ-

cient implementation methodology is presented. Ad-

417

Triantafyllou D. and A. Aspragathos N..

Deﬁnition and Classiﬁcation of Primitives for the Robotic Unfolding of a Piece of Clothing.

DOI: 10.5220/0005155704170422

In Proceedings of the International Conference on Knowledge Engineering and Ontology Development (KEOD-2014), pages 417-422

ISBN: 978-989-758-049-9

 2014 SCITEPRESS (Science and Technology Publications, Lda.)

ditionally, possible applications of the methodology

are proposed.

Particularly, in this paper, a dictionary of primi-

tives for critical features that deﬁne the conﬁguration

of a piece of clothing is introduced. The ”words” of

the dictionary include types of junctions, such as ”L”

and ”arrow” junctions, which constitute components

that are used in bibliography (Shapiro and Stockman,

2001) for perceiving 3D objects in 2D images. In-

spired by this approach, we use the same primitives to

deﬁne the conﬁguration of a piece of clothing. Nev-

ertheless, in the presented case the primitives do not

represent the joins of the facets of a 3D object but in-

dicate the existence of folds, overlappings, edges and

corners on a garment. Furthermore, machine vision

techniques are used to detect and classify these crit-

ical features into categories. The dictionary can be

used for any conﬁguration under the restriction that

it occurs from holding the garment in the air from its

outline and then laying it onto a table. The main dif-

ference from previous works that aimed at unfolding

clothes is that we try to analyse and comprehend the

conﬁguration of the garment no matter its shape. We

do not use predeﬁned templates, so even pieces of fab-

ric with unusual shapes can be handled.

The outline of this paper is as follows. Section 2

presents the dictionary of primitives in detail, while

in section 3 possible applications are described. In

section 4, the detection and classiﬁcation of the prim-

itives is implemented. Finally, section 5 includes ex-

perimental results and section 6 some conclusions and

ideas for future work.

2 INTRODUCTION OF

PRIMITIVES FOR THE CLOTH

UNFOLDING

The inﬁnite degrees of freedom of clothes result in

inﬁnite possible conﬁgurations. The introduced prim-

itives concern only conﬁgurations, where gravity is

used to convert crumbled clothes (Fig 1.b) into a more

straightened, crisp state. In particular, the accepted

conﬁgurations can occur after lifting the garment in

the air from a point at its outline when it is fully un-

folded (Fig 1.a), so that all major wrinkles are re-

moved, and then lay it on a working table, as in Fig

1.c. In this paper, conﬁgurations on a table are con-

sidered, although the differences when holding the

garment in the air are small. An easy way to de-

tect an outline point is to apply the lowest hanging

point technique as suggested in (Osawa et al., 2007),

(Cusumano-Towner et al., 2011). So, when a garment

grasped from a random point is hanging, its lowest

point is an outline point.

Figure 1: Conﬁgurations of the cloth: (a) the outline of the

cloth when it is fully unfolded, (b) a piece of clothing in a

crumbled conﬁguration, (c) the piece of clothing after it is

held in the air from its outline and laid on a working table.

Figure 2: Critical features for the unfolding task and their

characteristic points of intersection: (a) a fold, (b) an over-

lapping.

The primitives of the proposed dictionary are geo-

metric features that describe folds, overlappings, parts

of edges and corners on the surface of the lying piece

of clothing. We deﬁne as a fold the result of bending

a cloth over upon itself (Fig 2.a). On the other hand,

an overlapping occurs when a part of a cloth covers

partly another one, if the two participating parts are

not located next to each other in the garment’s un-

folded conﬁguration (Fig 2.b). To indicate the exis-

tence of a fold or overlapping, the detection of the

junction, where the participating layers intersect is

sufﬁcient (Fig 2). Furthermore, the term ”corners”

refers only to the corners that are located at the out-

line ( Fig 1.a)of the unfolded cloth.

The geometrical features, that constitute the in-

troduced primitives, are types of junctions which are

classiﬁed according to the number of edges and the

size of the angles they form into ”I”, ”L”, ”T” and

”arrow” junctions (Table 1). So, a fold is detected

by an ”arrow junction”, an overlapping is met when a

”T” junction occurs, a part of an edge is an ”I” junc-

KEOD2014-InternationalConferenceonKnowledgeEngineeringandOntologyDevelopment

418

Table 1: The primitive features for the unfolding task.

Features fold overlapping corner edge

Junction’s type ”arrow” junction ”T” junction ”L” junction ”I” junction

Junction’s type

Junction’s schema

tion, while the corners of the garment are indicated

by ”L” junctions. Furthermore, an ”L” junction could

also indicate a ”hidden fold”, which is actually a fold

whose folded part is hidden behind the visible part

of the garment. One way to distinguish a hidden fold

from a cloth’s corner is to lift the ”L” junction with the

robot and check if an ”arrow” junction was hidden at

the side of the cloth that was not visible. In addition,

the edges of a ”T” junction dictate which part of the

cloth is overlapped and which one overlaps the other

part. So, if we consider the ”T” junction as an in-

tersection of one edge to the middle of another edge,

then the part of the cloth that includes the ﬁrst edge

is overlapped by the part of the cloth with the second

edge.

The relationships described in the previous para-

graph cover a big part of the junctions that can be

found on a lying piece of clothing. Other types of

junctions could rarely occur in the conﬁgurations un-

der consideration thanks to gravity and for now are

not taken into account. In addition, we cannot ex-

clude the possibility that the types of junctions used as

primitives could occur from other circumstances than

the mentioned ones. Nonetheless, the reasonable lo-

cation on the cloth and the appropriate combination of

primitives can largely diminish false correspondences

between the primitives and their meaning. In section

3, apart from examples that prove that these mistakes

can be avoided, different ways that the primitives can

be appropriately connected and form layers are de-

scribed.

3 USES OF THE PRIMITIVES’

DICTIONARY

The introduced features can provide information for

the extraction of the garment’s layers in order to facil-

itate the unfolding task. The goal of this paragraph is

not to classify the layers based on the proposed primi-

tives, although it will be presented in future work. The

aim is to demonstrate the utility of the proposed dic-

tionary by showing how its primitives, detected in ap-

propriate combinations, can describe different types

of layers. So, in the same way that words put in an

appropriate order form sentences with different mean-

ings, the primitives of the proposed dictionary, de-

tected in an appropriate order, deﬁne layers that occur

from different conﬁgurations.

Figure 3: Primitives and their role to extracting edges for

the unfolding task: (a) extraction of a layer that can be un-

folded, (b) detection of an overlapped layer and a layer that

can be unfolded, (c) the effect of a ”loose fold”, (d) extrac-

tion of a layer when the cloth is in a half folded conﬁgura-

tion.

In Fig 3.a layer L can be deﬁned by a sequence

of the proposed primitives. Starting from one of the

arrow junctions (e.g. a1), which implies the existence

of a fold, and following its internal edge by detecting

”I” junction primitives, we result in the ”L” junction

c1, meaning a corner. Continuing in the same way,

arrow junction a2 is met and ﬁnally we result again to

arrow junction a1. This sequence of primitives sug-

gests the existence of a layer that can be unfolded,

since it does not include any ”T” junction that implies

overlapping. Similarly, in Fig 3.b, starting from arrow

junction a3 and following its internal edge we reach

a ”T” junction. The edge that led to the ”T” junction

implies that the layer that we started detecting (L2) is

overlapped by another layer (L1), as a result it can-

DefinitionandClassificationofPrimitivesfortheRoboticUnfoldingofaPieceofClothing

419

not be unfolded. On the other hand, the extraction of

the layer L1 can be conducted in a similar way as in

Fig 3.a, since, in this case, the ”T” junction suggests

that layer L1 overlaps another layer. So, depending

on the orientation of a ”T” junction different conclu-

sions about the unfoldability of a layer can occur. In

addition, in Fig3.c, the ”T” junction created by the

”loose fold” cannot affect the correct extraction of an

unfoldable layer since its position does not suggest

that is overlapped.

In addition, the primitives’ dictionary could also

be used for unfolding in situations that the garment

is brought in a half folded conﬁguration by using

the ”lowest hanging point” technique, as suggested

in (Osawa et al., 2007), (Cusumano-Towner et al.,

2011). The detection of the layer to unfold in addi-

tion to the detection of possible overlaps can be con-

ducted in a similar way as described in the previous

paragraph (Fig 3.d).

Finally, a scenario that could be investigated is the

detection of folds indicated by ”arrow” junctions, in

order to ﬁnd points at the outline of the garment un-

der unfolding so as to recognize the type of clothing

and unfold it as in (Kaneko and Kakikura, 2001). In

(Kaneko and Kakikura, 2001) the detection of such

points is made using the shadows created on the gar-

ment in combination with the convex shape created at

the outline of the hanging cloth near the fold. Since

folds do not always create sufﬁcient shadow for de-

tection we believe that the detection of ”arrow” junc-

tions can provide more candidates for outline points,

but this is left as future work.

4 THE DETECTION AND

CLASSIFICATION OF THE

PRIMITIVES

In this section a method for the implementation of the

primitives’ detection and classiﬁcation is presented.

The experiments are made on monochromatic clothes

using RGB images; nevertheless we have experimen-

tal evidence that the same procedure could be suc-

cessful on multicolour garments with a stereo camera.

The pictures are taken from a 20 cm distance from the

working table where the clothes are lying and no spe-

ciﬁc lighting conditions are used.

The ﬁrst step is to extract edges from the grayscale

picture depicting the primitive under detection. Bilat-

eral ﬁltering (Tomasi and Manduchi, 1998) , which

enhances the edges, is applied (the Gaussian window

is 20 pixels, the spatial domain standard deviation is

2, while intensity domain standard deviation is 0.6).

Figure 4: Examples of primitive features and their edge de-

tection: (a) a part of a towel’s edge, (b) the edge detection

of the towel’s edge, (c) a fold on a ﬂeece piece of clothing,

(d) the edge detection of the ﬂeece’s fold.

Figure 5: The detected edges from Ransac algorithm and

the clusters of their intersection points.

Then an automated procedure is used for a Canny de-

tector threshold that diminishes the noise, while ex-

tracting the edges. Particularly, the threshold is re-

duced until the ratio of the pixels that belong to the

edges and the noise to the rest pixels of the image be-

comes smaller than 0.025. In this way, while the ra-

tio is reduced, the noise pixels that are decreased are

more than the decreased edge pixels, since they are

enhanced by the bilateral ﬁlter. This procedure pro-

vides satisfactory edge detection for different kinds of

clothes regardless if their surface is smooth or rough

(e.g. towel). For example in Fig 4 primitives from

a towel and a ﬂeece piece of clothing are extracted

without particular noise.

To proceed, different edges are detected using

Ransac algorithm (Fig 5). Up to ﬁve edges are de-

tected in order to reduce computational cost and avoid

the detection of small edges created from noise. Sev-

eral of the detected edges might be double, meaning

KEOD2014-InternationalConferenceonKnowledgeEngineeringandOntologyDevelopment

420

Figure 6: Examples of the detection and classiﬁcation of primitives. In the ﬁrst row the grayscale images of the primitives

are presented while in the second row the corresponding detected features are depicted. For each image different edges are

depicted with different colors while the cluster that represents the ﬁnal point of intersection is marked with a blue circle. The

third row contains the results of the classiﬁcation.

that an edge and its shadow can be detected as two

different edges, that Ransac split an edge in two parts,

that the cloth has a thick hemline creating two edges

etc. Such examples can be seen in Fig 6. Further-

more, the detected edges that participate in a junction

usually do not intersect to only one point but within a

small area (cluster 1 in Fig 5). To detect this area the

points of the edges’ intersections are calculated and

clustered according to their distance from each other

using the single-link hierarchical clustering method

(the maximum relative distance of the cluster’s ele-

ments is 25 pixels). The cluster including the highest

number of elements is chosen as the junction of inter-

est (Fig 5).

Then, the number of the edges participating to

the junction is calculated. Double edges are uniﬁed,

based on their location and inclination, since they are

almost parallel and close to each other (eg in Fig 5

edge 3 and 4). Once the junction of interest and the

number of intersecting edges is found, the angles cre-

ated by the edges of the junction are measured to de-

termine the type of junction. So, according to its ge-

ometry, the junction is classiﬁed to one of the primi-

tives deﬁned in section 2.

5 EXPERIMENTAL RESULTS

The detection and classiﬁcation algorithm was tested

in 40 different samples using 4 different monochro-

matic fabrics. In 80% of the cases, the junctions were

detected and classiﬁed into the correct category of

primitives. Examples depicting the correct detection

and the classiﬁcation of the primitives can be seen in

Fig 6. On the other hand, in 7, 5% the arrow junc-

tions were mistaken as other type of junction because

of the big curvature of the bended layer of fabric near

the junction and the shadow created (Fig. 7.a). With

straightening movements made by the robot, this cur-

vature could be diminished and better results would

be obtained. Furthermore, in 7.5% the noise created

problems extracting wrong edges and in 5% the junc-

tion was near the frame of the picture, so one of the

participating edges was not big enough to be detected

(Fig. 7.b). The additional evaluation of a second pic-

ture, slightly dislocated from the ﬁrst one, is consid-

ered as a measure that could enhance the results and

will be investigated in the future.

Figure 7: Examples where the method failed to categorize

correctly the primitives: (a) the folded part creates shade be-

cause of the big curvature near the point the fold is formed,

(b) one of the edges participating to the arrow junction is

not big enough to be detected.

DefinitionandClassificationofPrimitivesfortheRoboticUnfoldingofaPieceofClothing

421

6 CONCLUSION

In this paper, a dictionary of primitives, that deﬁne

critical features on a piece of clothing for its unfold-

ing, is proposed. The dictionary can be used for

pieces of clothing and fabrics of various shapes, since

we do not use restricting predeﬁned templates. The

primitives are detected and classiﬁed based on geo-

metrical criteria.

The combination of primitives can represent dif-

ferent conﬁgurations of a garment laid properly on

a table. Layers that can be unfolded, folds, over-

lappings and corners of the garment can be detected

and used for planning manipulations for the unfolding

task. The detection and classiﬁcation of the primitives

is implemented using machine vision techniques. Re-

sults show the effectiveness of the proposed approach.

As future work, the implementation of layers’ ex-

traction and manipulation in order to unfold a piece

of clothing is planned. Additionally, a scenario to be

investigated is the detection of folds, in order to ﬁnd

points at the outline of the garment under unfolding

so as to recognize the type of clothing and unfold it.

REFERENCES

Cusumano-Towner, M., Singh, A., Miller, S., Brien, J. O.,

and Abbeel, P. (2011). Bringing clothing into desired

conﬁgurations with limited perception. In IEEE Inter-

national Conference on Robotics and Automation.

Han, F. and Zhu, S. (2005). Cloth representation by shape

from shading with shading primitives. In Conference

in Computer Vision and Pattern Recognition.

Kaneko, M. and Kakikura, M. (2001). Planning strategy for

putting away laundry - isolating and unfolding task. In

IEEE International Symposium on Assembly and Task

Planning.

Maitin-Shepard, J., Cusumano-Tower, M., Lei, J., and

Abbeel, P. (2010). Cloth grasp point detection based

on multiple-view geometric cues with application to

robotic towel folding. In IEEE International Confer-

ence on Robotics and Automation.

Osawa, F., Seki, H., and Kamiya, Y. (2007). Unfolding of

massive laundry and classiﬁcation types by dual ma-

nipulator. In Journal of Advanced Computational In-

telligence and Intelligent Informatics.

Shapiro, L. and Stockman, G. (2001). Computer Vision.

Prentice Hall, 1st edition.

Tomasi, C. and Manduchi, R. (1998). Bilateral ﬁltering for

gray and colour images. In IEEE International Con-

ference on Computer Vision.

Triantafyllou, D. and Aspragathos, N. A. (2011). A vision

system for the unfolding of highly non-rigid objects

on a table by a manipulator. In International Confer-

ence on Intelligent Robotics and Applications.

Yamazaki, K. and Inaba, M. (2009). A cloth detection

method on image wrinkle feature for daily assistive

robots. In Conference on Machine Vision Applica-

tions.

Yamazaki, K., Nagahama, K., and Inaba, M. (2011). Daily

clothes observation from visible surfaces based on

wrinkle and cloth-overlap detection. In Conference

on Machine Vision Applications.

KEOD2014-InternationalConferenceonKnowledgeEngineeringandOntologyDevelopment

422