Motion Curved Surface Analysis and Composite for Skill Succession

using RGBD Camera

Kaoru Mitsuhashi

, Hiroshi Hashimoto

and Yasuhiro Ohyama

Department of Mechanical engineering, School of Engineering, Tokyo University of Technology, Hachioji, Tokyo, Japan

Master Program of Innovation for Design and Engineering, Advanced Institute of Industrial Technology, Tokyo, Japan

Keywords: Skill Succession, Microsoft Kinect, B-spline Curve Surface, Visualization, Gradient Curvature Distribution,

Experts and Beginners, Composite Surface, Skeleton and Curved Surface Succeeding Method.

Abstract: The skill succession method is almost oral. It is not quantitative but qualitative. Quantitative succession is

difficult. In this research, after tracking of a subject's motion using RGBD camera, a subject's motion is

visualized as the motion curved surface. Expert and beginner perform the sports and entertainment motion,

and the character of the surface is analyzed. The character is the maximum curvature and surface area. In

addition, we suggest the composite surface, because one RGBD camera is not all tracking motion by occluding

the obstacle or subject’ body parts. Finally, we confirm the validity of skill succession by watching skeleton

motion movie and curved surface.

1 INTRODUCTION

Beginner trains to watch and imitate the expert

behavior, in the entertainment of the Noh play and

Kabuki play, and sports play, engineering of

operating a machine (Hashimoto et al., 2011).

However, their skill succession/teaching are

qualitative, not quantitative. They are expressions in

abstract languages, such as onomatopoeia, or

metaphor of an object image, and the quantitative

evaluation is difficult to express and perform

(Hasegawa and Fukumura, 1996). Therefore, the skill

succession/teaching cannot be confirmed, the same

behavior is not always repeated.

Then, an expert's motion is captured by video

camera photography, and the motions are analyzed in

research or software (Takeo and Natsu, 2011),

(Cheung et al., 2003; Sigal and Black, 2006). The

method is the motion capture by one or more camera

sets, with the background subtraction technique,

extracts a human's outline/marker joints and displays

only a human's motion. The motion can be preserved,

and the reproducibility is high. However the

extraction of human position is difficult, and

quantitative evaluation is limited or no meaning.

Furthermore, human joints are needed to capture

equipped markers, by forcing marker wearing on a

subject. Therefore, we can hardly expect to track the

usual motion.

We focus Microsoft Kinect, which is a reasonable

and easy operation, and capture the motion using it.

Kinect can recognize pictures and depth positions,

which is a useful tool function and expected the

application to three-dimensional measurement.

Kinect can extract a human's outline, and the position

of the human skeletons and joints. In the conventional

research, angles of the skeleton and joint positions are

measured (Murao et al., 2011; Hashimoto et al.,

2014). In addition, we visualize a human joint

trajectory of motion into a curved surface (it is called

a motion curved surface), and we extract the

difference between beginners and experts from the

form or curvature of the motion curved surface in

previous research (Mitsuhashi et al., 2014; Suneya et

al., 2014). Therefore, we can evaluate technical skill

quantitatively, and except the skill

succession/teaching for expert’s skill easily.

However, some joints are not tracked by occluding an

obstacle or body parts (joints, body, arm, leg, head...),

(Deutscher et al., 2000) because tracking view is one

direction (one Kinect). Therefore, the motion curved

surface is lacked by the occlusion. On the other hand,

we have never confirmed the validity of skill

succession using a motion curved surface.

In this research, we compose the motion curved

surfaces made from the multiple Kinect view, so as to

track the whole joint motion in more detail. Moreover

406

Mitsuhashi K., Hashimoto H. and Ohyama Y..

Motion Curved Surface Analysis and Composite for Skill Succession using RGBD Camera.

DOI: 10.5220/0005569904060413

In Proceedings of the 12th International Conference on Informatics in Control, Automation and Robotics (ICINCO-2015), pages 406-413

ISBN: 978-989-758-123-6

 2015 SCITEPRESS (Science and Technology Publications, Lda.)

we investigate the effectiveness of the composite

motion curved surface. In addition, we confirm the

validity of skill succession by watching skeleton

motion movie and curved surface.

2 EXPERIMENT METHOD

2.1 Motion Tracking Method

In this paper, we track the motion of human’s joints

in drawing gesture expression using Kinect. A user

expresses object shape by moving the right hand, left

hand, or both hands with depth sensing and image

recognition. Figure 1 shows the tracking situation.

Kinect is placed the height position of 1.75m and the

distance between Kinect and a user is 2.0m. Figure 2

shows an image recognition of the user. Figure 2(a)

shows an image recognition of the user. Figure 2(b)

shows a depth image recognition with human joints

and skeleton model. Position of joints and skeletons

is estimated by Kinect driver. In this paper, we

measure the position of a right hand, right elbow,

right shoulder, left hand, left elbow, left shoulder, and

neck. Line segments by gesture are displayed with

measuring the position of the hand (right or left hand)

using the OpenCV library. Line segments by gesture

are displayed with measuring the position of the hand

(right or left hand) using OpenCV library. Kinect

programing language is C/C++ and using openNI2,

Figure 1: Motion tracking situation.

(a) RGB color image (b) Depth image

Figure 2: Kinect view.

NiTE2 library. A user’s motion is tracked in every

0.02 second, and the measured position is placed with

the time series.

2.2 Curved Surface Visualization

The subject's motion is visualized to a curved surface

in the preceding section. In order to visualize a curved

surface, the data of a subject's joint position of point

cloud based on a time series is preserved, and B-

spline curved surface is fitted to the point cloud by the

approximation. The curved surface makes a subject's

trajectory the direction of u, and makes joint positions

the direction of v. Figure 3 shows motion, trajectory

and curved surface when the subject opens the arms

and squats down. The generated curved surface

calculates the size of a curved surface, normal vectors,

tangent vectors, and curvatures using 3D-CAD

software Rhinoceros in Figure 3. Figure 3(b) shows

the trajectory of upper joints. Figure 3(c) shows the

visualized curved surface. Figure 3(d) shows the

gradation display of curvature and Figure 3(e) the

zebra mapping display. Zebra mapping is an

analytical technique to visualize continuities of the

curvature.

(a) RGB color image

(b) Joints trajectory (c) Visualized surface

(d) Zebra mapping (e) Gradient curvature

Figure 3: Visualized motion curved surface.

However, if all the motions are transferred the

curved surface display, a curved surface will be

Start

End

MotionCurvedSurfaceAnalysisandCompositeforSkillSuccessionusingRGBDCamera

407

twisted or overlapped. Then, tangent and normal

vectors are calculated, and the first standard normal

and tangent vectors are decided. And a curved surface

is divided if the angle between the standard vector and

the other is larger than 180 degrees. Furthermore, a

curved surface is divided also if the self-intersection

on a curved surface or edge is occurring. Then, we are

able to prevent a twist and overlap of a curved surface.

3 ANALYSIS OF SKILL MOTION

3.1 Throw Motion in Darts

We investigate the difference of throwing motion in

Darts between 5-year-experience expert and

beginner. Subjects perform the darts motion in the

front of Kinect, their upper half of the body (hand,

elbow, and shoulder) is tracked. The situation of

throwing motion in darts is shown in Figure 4. Figure

4 (a) shows expert’s motion of RGB color image and

depth image with joints and skeletons, and Figure 4

(b) shows beginner’s motion. The visualized curved

surface of the expert’s motion is shown in Figure 5.

Figure 5 (a) shows the curved surface with the

gradient curvature distribution when a right arm is

throwing and Figure 5 (b) shows the curved surface

with zebra mapping. The visualized curved surface of

the beginner’s motion is shown in Figure 6. The

curved surface is divided hand-elbow and elbow-

shoulder, because there is the self-intersection in

Figure 6. Figure 6 (a) shows the curved surface with

gradient curvature distribution when a right arm is

throwing and Figure 6 (b) shows the curved surface

with zebra mapping. In this result, the expert’s motion

(a) Expert (mirror image)

(b) Beginner (mirror image)

Figure 4: Throwing motion in darts.

(a) Zebra mapping (b) Gradient curvature

Figure 5: Motion curved surface of expert in darts.

(a) Zebra mapping (b) Gradient curvature

Figure 6: Motion curved surface of beginner in darts.

Table 1: Maximum curvature, area, and shape in darts.

surface is fan shaped, the elbow is fixed. Namely, the

expert’s hand trajectory is curved. Expert turns his

hand toward the target in the finish. According to an

expert’s opinion, fixing the elbow is most important.

On the other hand, the beginner’s surface is triangle

shape. Namely, the beginner’s hand trajectory is

straight. From Figure 5 and Figure 6, curved surface

of expert’s motion had more flat parts than the

beginner’s motion on the whole. This result is the

same in zebra mapping. The striped zebra pattern of

the beginner’s motion is heterogeneous. As

mentioned above, the measuring result of the

maximum curvature and the curved surface area is

shown in Table 1. From Table 1, the expert’s surface

of change of curvature is focally larger than the

beginner’s surface. Beginner’s surface at start is

largest curvature. On the other hand, the expert’s

surface area is similar to the beginner’s area.

3.2 Leg Raising Motion in Stretch

We investigate the difference in leg raising motion

(side plank) of Stretch 3-year-experience expert and

beginner. Subjects perform the darts motion in the

front of Kinect, their lower half of the body (foot,

Maximum

curvature

[rad/mm]

Area

]

Shape

Expert 0.91 0.18

Fan shape

Beginner 0.24 0.12

Triangle

ICINCO2015-12thInternationalConferenceonInformaticsinControl,AutomationandRobotics

408

knee, and hip) is tracked. Side plank is a motion that

the whole body is supported using the arm and leg,

and opposite leg is raising up and down. The situation

of throwing motion in side plank is shown in Figure

7. Figure 7 (a) shows expert’s motion of RGB color

image and depth image with joints and skeletons, and

Figure 7 (b) shows beginner’s motion. Figure 8 shows

the multiple curved surfaces of whole body in leg

raising motion. In Figure 8, we need to focus the

lower half of the body. Figure 9 (a) shows the curved

surface with the gradient curvature distribution when

a right leg of expert is raising and Figure 9 (b) shows

the curved surface with zebra mapping. Figure 10 (a)

shows the curved surface with the gradient curvature

distribution when a right leg of beginner is raising and

Figure 10 (b) shows the curved surface with zebra

mapping. From figures, the stripped zebra pattern of

expert is along the curved lines, but beginner’s pattern

is not. As mentioned above, the measuring result of

the maximum curvature and the curved surface area

is shown in Table 2. From Table 2, the expert’s

surface of change of curvature is focally larger than

the beginner’s surface. On the other hand, the expert’s

surface area is larger than beginner’s area.

3.3 Evaluation of Curved Surface

Table 3 shows the maximum curvature, area, and

shape of motion curved surface in swimming the

crawl, karate thrusts (Mitsuhashi, Ohyama, and

Hashimoto, 2014), darts throw, side plank, and

calligraphy. From Table 3, we find the expert’s

maximum curvature and area is larger than

(a) Expert

(b) Beginner

Figure 7: Leg raising motion in side plank.

Figure 8: Motion curved surfaces of body in side plank.

(a) Zebra mapping (b) Gradient curvature

Figure 9: Motion curved surface of expert in side plank.

(a) Zebra mapping (b) Gradient curvature

Figure 10: Motion curved surface of beginner in side plank.

Table 2: Maximum curvature, area, and shape in side plank.

Table 3: Maximum curvature, area, and shape

of the motion curved surfaces.

beginner’s. Additionally the expert’s shape is

complicated, the beginner’s shape is flat. Therefore,

we need to focus the maximum curvature, area, and

shape, and their outputs or results are also important.

Maximum

curvature

[rad/mm]

Area

]

Shape

Expert 0.87 0.29

Triangle

Beginner 0.21 0.13

Rectangle

Maximum

curvature

[rad/mm]

Area

]

Shape

Expert 5.60 0.19 Concave

Beginner 0.82 0.43 Convex

Expert 10.63 0.21 Thin

Beginner 1.74 0.66 Thick

Expert 0.91 0.18 Fan shape

Beginner 0.24 0.12 Triangle

Expert 0.87 0.29 Triangle

Beginner 0.21 0.13 Rectangle

Expert 0.46 0.15 Projection

Beginner 0.24 0.10 Flat

Swimming

crawl

Karate

thrust

Darts

throw

Side plank

leg raise

Caligraphy

MotionCurvedSurfaceAnalysisandCompositeforSkillSuccessionusingRGBDCamera

409

4 COMPOSITE CURVED

SURFACE

4.1 Limitations of One Kinect View

Body joints are tracked by one Kinect until preceding

chapter, for that reason, some joints are not tracked

by occluding an obstacle or body parts oneself. In

addition, the motion curved surfaces are different

shape from the front view and back view. Because the

motion surface exist both the correct tracking parts

and incorrect tracking parts. Therefore the tracking all

motion in large or rotating motion is difficult for one

Kinect. Then, we suggest to track the motion from the

multiple direction using the multiple Kinect, the

multiple curved surfaces are composited. Figure 11

shows the surface composition method. One correct

surface and the other correct surface are composited,

and the incorrect surfaces are ignored from Figure 11.

Figure 12 shows the tracking situation of multiple

Kinect. The composite curved surface is made with 2

Kinect in this paper.

Figure 11: Schematic of composite surface.

Figure 12: Tracking situation of multiple Kinect.

4.2 Validity of Composite Curved

Surface

Throw in darts in preceding chapter is tracked using

2 Kinect, one Kinect is tracking from front view

position to 1.50m distance, and the other is from a

side view position to 1.50m distance. Because we

confirm the validity of the composite curved surface.

Figure 13 (a) shows the motion curved surface from

side view, Figure 13 (b) shows from front view. The

curved surface of Figure 13 (a) exists the incorrect

parts at the start and end and the correct parts in hand

trajectory, because the depth sensor is less exact than

image processing. On the other hand, the motion

curved surface of Figure 13 (b) exists the correct parts

in all motion. Then, the Edge 2 and Edge 3 are

changed to the Edge 5 and Edge 6, watching the RGB

image view. In this result, the composite surface is

similar to the motion surface from front view. Figure

13 (c) shows the composite curved surface.

Therefore, we prove the composite surface from

multiple view valid.

(a) Left view (b) Front view

Figure 13: Composite motion curved surface of expert in

darts throwing motion.

4.3 Swing Motion in Tennis

Large and rotating motion is tracked using the

composite curved surface method. It is a swing

motion in tennis. Some joints are occluded by the

arms of oneself at start and end. Figure 14 shows the

situation of multiple Kinect for rotating motion in

tennis. One Kinect is placed in the front, the other is

placed in the back. Figure 15 shows the motion

curved surface from top view using multiple Kinect.

Figure 15 (a) shows from back view, Figure 15 (b)

shows from front view. The curved surface of Figure

15 (a) exists the correct parts in Edge 1 and Edge 2 (at

the start) and the incorrect parts at the end by

occluding the arm. On the other hand, the motion

curved surface of Figure 15 (b) exists the correct parts

in Edge 3 (at the end) and the incorrect parts at the

ICINCO2015-12thInternationalConferenceonInformaticsinControl,AutomationandRobotics

410

(a) Perspective view

(b) Back view (c) Front view

Figure 14: Situation of swing motion in tennis.

(a) Back view (b) Front view

Figure 15: Composite motion curved surface of expert in

darts throwing motion.

start. Then, the composite surface is made from

selected the Edge 1, Edge 2, and Edge 5 watching the

RGB image view. Figure 15 (c) shows the composite

curved surface. From Figure 15(c), expert’s motion

surface from top view is more smooth and concave

than the beginner’s surface. Then, the measuring

result of the maximum curvature and the curved

surface area is shown in Table 4. From Table 4, the

expert’s surface of change of curvature is focally

larger than the beginner’s surface. On the other hand,

the expert’s surface area is larger than the beginner’s

area. Therefore, we can make the motion curved

surface and the difference of the expert and beginner

from large and rotating motion also.

Table 4: Maximum curvature, area, and shape of

composite surface in tennis.

5 SKILL SUCCESSION USING

MOTION CUVED SURFACE

We have never succeeded using a motion curved

surface, only confirmed the difference of the expert

and beginner in a motion curved surface character.

The skill succession method is watching a skeleton

motion (animation) and curved surface. The

animation is 3D viewer. It can rotate and translate the

view position and focal position. The animation

program is written by C++ and openGL library, and

it read the joint data. Figure 16 shows the skeleton

movie and motion curved surface of right upper half

of the body in swimming the crawl. Then, we confirm

the validity of skill succession using a motion curved

surface. First, two beginner subjects are tracked the

throw motion in darts. Next, one subject watch only

the motion movie, and the movie are expert’s motion.

The other watch the skeletons motion (animation) and

curved surface, the skeleton motion (animation) is the

expert’s or subject oneself. And both subjects train to

watch or move in 10 minutes. Figure 17 (a) shows the

watching only movie and moving situation, Figure 17

(b) shows the skeleton motion and curved surface of

right upper half of body (hand, elbow, and shoulder).

After training, two beginner subjects are tracked the

throw motion in darts again

Figure 16: Skeleton animation and motion curved surface

in swimming the crawl.

Maximum

curvature

[rad/mm]

Area

]

Shape

Expert 2.40 1.84

Concave

Beginner 1.30 1.16

Flat

MotionCurvedSurfaceAnalysisandCompositeforSkillSuccessionusingRGBDCamera

411

(a) Watching video (b) Motion with surface

Figure 17: Training the throwing motion in darts.

Figure 18 shows the before training curved

surface for two beginner. They are triangle shape or

exist straight trajectory of hand, and two subjects

don’t fix the elbow. Figure 19 shows the after training

curved surface. Figure 19 (a) shows the only watch

motion movie, Figure 19 (b) shows the skeleton

motion and curved surface. From figures, the surface

of Figure 19 (a) remain the same shape. On the other

hand, the surface of Figure 19 (b) is similar to expert’s

fan-shaped surface, and his trajectory is a curved line

(segment). Table 5 shows the maximum of curvature

and area for expert, only movie training, and skeleton

and curved surface training. The skeleton and curved

surface trainer’s motion are similar to expert’s motion

from Table 5. In this result, we prove the skill

succession using the motion curved surface valid.

(a) Watching video (b) Motion with surface

Figure 18: Motion curved surface of before training.

(a) Watching video (b) Motion with surface

Figure 19: Motion curved surface of after training.

6 CONCLUSIONS

In this research, expert and beginner perform the

sports and entertainment motion for skill succession,

Table 5: Maximum curvature, area, and shape

throwing motion training in darts.

and the character of the surface is analyzed.

Additionally we combine the motion curved surfaces

made from the multiple Kinect view, so as to track the

whole joint motion in more detail. We investigate the

effectiveness of the composite motion curved surface.

In addition, we confirm the validity of skill

succession by watching skeleton motion movie and

curved surface. In this result, we find the expert’s

maximum curvature and area is larger than

beginner’s. Moreover the expert’s shape is

complicated, the beginner’s shape is flat. In addition,

we can make the composite motion curved surface

from multiple Kinect, and the difference of the expert

and beginner from large and rotating motion also.

Finally, we prove the skill succession using the

skeleton animation and motion curved surface valid.

In future work, the database is constructed the other

motion curved surface, and the motion velocity or

acceleration curved surface is suggested. In addition,

the new method of skill succession using motion

curved surface.

ACKNOWLEDGEMENTS

This work was in part supported by JST RISTEX

Service Science, Solutions and Foundation Integrated

Research Program and JSPS KAKENHI Grant

Number 60208460.

REFERENCES

Hashimoto, H., Yoshida, I., Teramoto, Y., Tabata, H., and

Han, C., 2011. Extraction of Tacit Knowledge as Expert

Engineer's Skill based on Mixed Human Sensing,

Proceedings of the 20th IEEE International Symposium

on Robot and Human Interactive Communication,

pp.413-418, 2011.

Taki, T., Hasegawa, J. I., & Fukumura, T. (1996,

September). Development of motion analysis system for

quantitative evaluation of teamwork in soccer games.

In Image Processing, 1996. Proceedings, International

Conference on Vol. 3, pp. 815-818. IEEE.

Maximum

curvature

[rad/mm]

Area

]

Shape

0.91 0.18 Fan shape

Before 0.26 0.15 Triangle

After 0.47 0.16 Triangle

Before 0.53 0.13 Triangle

After 0.82 0.17 Fan shape

Expert

Only movie

Skeleton and

Surface

Beginner

ICINCO2015-12thInternationalConferenceonInformaticsinControl,AutomationandRobotics

412

Yasushi Takeo ， Wataru Natsu, 2011. Development of

valuation Method for Measurement Skill Training,

Proceedings of International Symposium on

Standardization Education and Research 2011 Tokyo

Japan, pp.130-145.

Cheung, K. M. G., Simon Baker, and Takeo Kanade., 2003.

Shape-from-silhouette of articulated objects and its use

for human body kinematics estimation and motion

capture, Computer Vision and Pattern Recognition,

2003. Proceedings. 2003 IEEE Computer Society

Conference on. Vol. 1. IEEE, pp.77-84.

Sigal, Leonid, and Michael J. Black., 2006. Humaneva:

Synchronized video and motion capture dataset for

evaluation of articulated human motion. Brown

Univertsity TR 120.

Toshiyuki Murao, Yasuyuki Hirao and Hiroshi Hashimoto,

2011. Skill Level Evaluation for Taijiquan based on

Curve Fitting and Logarithmic Distribution Diagram of

Curvature, SICE Journal of Control, Measurement, and

System Integration, 4, 1, pp.001–005.

Hiroshi Hashimoto, Mizuki Nakajima, Seiichi Kawata,

Sang-Gyu Shin, Toshiyuki Murao, 2014. Skill Level

Evaluation of Taijiquan based on 3D Body Motion

Analysis, Proc. Of IEEE International Conference on

Industrial Technology (ICIT2014), CD-ROM.

Kaoru Mitsuhashi, Hiroshi Hashimoto, and Yasuhiro

Ohyama, 2014. The Curved Surface Visualization of the

Expert Behavior for Skill Transfer Using Microsoft

Kinect, 11th International Conference on Informatics in

Control, Automation and Robotics (ICINCO 2014),

September1-3, Wien, Austria, pp.550-555.

Mitsuki Suneya, Masaki Kutsuna, Kaoru Mitsuhashi,

Yasuhiro Ohyama, and Hiroshi Hashimoto, 2014. The

Curved Surface Visualization of Behaviors for Skill

Inheriting using RGBD camera. SICE System

Integration 2014 (SI2014), December 14-17, Tokyo,

Japan. Pp.67-71, (in Japanese).

Deutscher, Jonathan, Blake, Andrew, Reid, Ian, 2000.

Articulated body motion capture by annealed particle

filtering. In: Computer Vision and Pattern Recognition,

2000. Proceedings. IEEE Conference on. IEEE, pp.126-

133.

MotionCurvedSurfaceAnalysisandCompositeforSkillSuccessionusingRGBDCamera

413