Skill Level Evaluation of Motion Curved Surface Character

Kaoru Mitsuhashi

, Hiroshi Hashimoto

and Yasuhiro Ohyama

Department of Mechanical Engineering, School of Engineering, Tokyo University of Technology, Hachioji, Tokyo, Japan

Master Program of Innovation for Design and Engineering, Advanced Institute of Industrial Technology, Tokyo, Japan

Keywords: Skill Level, Microsoft Kinect, B-spline Curve Surface, Visualization, Gradient Curvature Distribution,

Experts and Beginners, Correlation Diagram, Motion Curved Surface Training.

Abstract: The skill teaching/succession method is not quantitative but qualitative, which is abstract oral or gesture

expression. Quantitative teaching is difficult for teacher/instructor. In previous research, Expert and beginner

perform the sports and entertainment motion, and the character of the motion curved surface is analysed using

Microsoft Kinect (RGBD camera). The character is the maximum curvature and surface area. However, the

usage of characters is uncertain. In this research, we investigate the correlation of maximum curvature and

surface area from motion curved surface in before and after training. Therefore, we visualize the different

correlation of experts and beginners from the characters and the transition of the skill training.

1 INTRODUCTION

The physical motion of experts, in the entertainment,

traditional ceremony, sport, engineering, is difficult

to play for beginners. Then, learners/beginners are

taught the skill by expert teacher/instructor, and are

training repeatedly. The training method is watching

and imitating the expert physical motions (Hashimoto

et al., 2011). However, the teaching method is still not

quantitative but qualitative, which are expressions in

abstract words, onomatopoeia words, or metaphor

(Fujino et al., 2005). The quantitative skill teaching is

difficult to express and perform (Taki et al., 1996).

Therefore, beginners cannot always imitate the same

motion because of the different recognition from

beginners (learners) and experts (teachers).

In conventional research, physical movements are

captured and analyzed by multiple video camera

movie and application (Takeo and Natsu, 2011),

(Cheung et al., 2003), (Sigal and Black, 2006).

However, the capture of physical movement is

difficult in equipment, which should be wearing the

many markers and installing the large space.

Furthermore, only the movie evaluation is limited or

no meaning. Therefore, the physical motion (of

experts and beginner) is evaluated just a little. On the

other hand, we focus Microsoft Kinect, which is a

reasonable and easy operation/equipment. Kinect can

recognize pictures and depth positions, and is a useful

tool function and expected the application to three-

dimensional (3D) measurement. Kinect can extract a

human's outline and the position of the human

skeletons/joints automatically. Then, angles of the

skeleton and joint positions are measured (Murao et

al., 2011), (Hashimoto et al., 2014). However, they

can only evaluate the joint angle in time, but cannot

invest the whole body motion. Moreover, the only

quantitative evaluation of joint angle and extracting

position may be no meaning. Namely, only joint

angle evaluation is not necessary in many cases.

In previous research, we visualize a physical

motion (human joint trajectory) into a motion curved

surface, and extract the difference between beginners

and experts (Mitsuhashi et al., 2014), (Suneya et al.,

2014). Therefore, we can evaluate physical/technical

skill quantitatively, and suggest the skill

succession/teaching method for expert

teacher/instructor. In addition, we compose the

motion curved surfaces made from the multiple

Kinect view, so as to track the whole joint motion in

more detail, and confirm the validity of skill

succession by watching skeleton motion movie and

curved surface (Mitsuhashi et al., 2015). However,

the exemplary motion curved surface has not been yet

established for physical/technical motion, because the

number of subjects is very few. Only the visualizing

motion curved surface for expert instructor cannot be

evaluated the skill level or the exemplary motion.

Therefore, the large number of subjects and

numerical tendency for motion curved surface is

Mitsuhashi, K., Hashimoto, H. and Ohyama, Y.

Skill Level Evaluation of Motion Curved Surface Character.

DOI: 10.5220/0006004304990504

In Proceedings of the 13th International Conference on Informatics in Control, Automation and Robotics (ICINCO 2016) - Volume 2, pages 499-504

ISBN: 978-989-758-198-4

499

necessary for skill evaluation.

In this paper, we create many motion curved

surfaces using the previous method, and investigate

the correlation of character from the motion curved

surface. The character is maximum curvature and

surface area. By means of expressing the diagram of

the motion curved surface character, the different and

tendency of experts and beginners is extracted. In

addition, we investigate the transition of training

effectiveness for beginners from correlation diagram.

2 EXPERIMENT METHOD

2.1 Motion Tracking Method

In this paper, the movement of the whole body in

physical motion is tracked by Kinect. The trajectories

of human’s joints are measured by depth sensing and

image recognition. Figure 1 shows the motion

tracking state. Multiple Kinects are placed the height

position of 1.0-1.75m and the distances between

Kinects and human are 1.5-2.0m. We measured using

multiple Kinect, because some joints are not tracked

by occluding an obstacle or body parts oneself in

large or rotating motion. In addition, the motion

curved surfaces are different shape from the front

view and back view, because the motion curved

surface exist both the correct tracking parts and

incorrect tracking parts.

Figure 1: Motion tracking state.

(a) Depth image (b) RGB color image

Figure 2: Kinect view.

Then, the multiple curved surfaces are composited

from both correct curved surfaces. Figure 2 shows an

image recognition of the user. Figure 2(a) shows an

image recognition of the human. Figure 2(b) shows a

depth image recognition with human joints and

skeleton model. Positions of joints and skeletons are

estimated by Kinect driver. Kinect can measure the

position of 25 joints. Line segments in Figure 2(b) are

displayed with measuring the position of the joints

using the OpenCV library. Kinect programing

language is C/C++ and using openNI2, NiTE2

library. A human’s joint positions are

measured/calculated in every 0.02 seconds with the

time series.

2.2 Motion Curved Surface

Visualization

The physical motion is visualized to a curved surface

in the preceding section. In order to visualize a curved

surface, the data of a human joint position of point

cloud based on a time series is preserved, and B-

spline curved surface is fitted to the point cloud by the

approximation. The curved surface makes the human

trajectory direction in time series, and makes the

direction of joint positions, which is hand-to-elbow,

elbow-to-shoulder, etc... Figure 3 shows the joint

trajectories and motion curved surface when the

human opens the arms and squats down. The

generated curved surface calculates the area, size,

normal vectors, tangent vectors, and curvatures using

3D-CAD software Rhinoceros in Figure 3. Figure

3(b) shows the trajectory of upper joints. Figure 3(c)

shows the gradation display of curvature and Figure

3(d) shows the zebra mapping display. Zebra

mapping is an analytical technique to visualize

continuities of the curvature.

(a) RGB color image (b) Joints trajectory

Figure 3: Visualized motion curved surface.

ICINCO 2016 - 13th International Conference on Informatics in Control, Automation and Robotics

500

The motion curved surface is fitted by an

approximation method. The lines are only continuous

segments because the joint trajectories are discrete

point cloud; that is, the lines are not enough to create

curved lines. Then, the point cloud is converted to

fitting curve lines by approximation methods. The

approximation is the method for smoothly passing a

curved line or surface through only the neighborhood

of the point cloud, not through all the points. It

enables to control fuzzily the occurrence of the gap

and vibration of the joint trajectories by the error of

the sensor or image recognition. In this paper, we

adopt the approximation method, and the uniform

cubic B-spline curved line or surface. B-spline

surface allows for a singular point and maintaining

the curvature continuity.

When the physical motions are converted the

motion curved surface display, a curved surface may

be twisted or overlapped. Then, the motion curved

surface is divided if the angle between the standard

vector and the other is larger than 180 degrees, after

the first standard normal and tangent vectors are

decided. Furthermore, the motion curved surface is

divided also if the self-intersection on a curved

surface or edge is occurring. Then, we are able to

prevent a twist and overlap of a curved surface.

3 MOTION CURVED SURFACE

OF SOCCER KICK

3.1 Motion Curved Surface Shape

We investigate the physical motion curved surfaces

of the inside (pass) kick in soccer. The subjects are

10-year-experience expert and beginner, the number

of experts is 3, beginner is 17. Subjects perform the

kicking motion in the front and side of Kinects, their

lower half of the body (foot, knee, and hip) is tracked.

All subjects are measured two times every one week.

The situation of inside kick motion in soccer is shown

in Figure 4. From figure, the subjects kick in imaging

pass far away.

Figure 4 (a) shows expert’s motion of RGB and

depth image with joints and skeletons, and Figure 4

(b) shows beginner’s motion. The visualized physical

motion curved surface of the expert’s is shown in

Figure 5. Figure 5 shows the motion curved surface

with the gradient curvature distribution when the right

leg is kicking. In this result, second time motion is not

different from first motion. From the figures, the

expert’s motion curved surfaces have fanned shape,

because the expert’s hip motion is small and the foot

trajectory is a conic arc (the foot motion is large).

Three expert’s surfaces (motions) are similar to other

expert’s surface (motion). These results are the same

in zebra mapping.

Figure 6 shows the beginner’s motion curved

surfaces with the gradient curvature distribution when

the right leg is kicking. In this result, the beginner’s

motion curved surfaces have trapezoid or rectangular

shape, because the beginner’s hip motion is large and

the foot trajectory is a straight line (the foot motion is

small). All surfaces (motions) are not similar to other

surfaces (motions). According to an expert’s opinion,

the pass ball control is bad, if the hip motion is large.

(a) Expert image

(b) Beginner image

Figure 4: Inside kick motion in soccer.

(a) 1st time (b) 2nd time

Figure 5: Motion curved surface of expert

(gradient of curvature in inside kick).

In addition, stretching nee and fixing hip (waist)

is most important. Learner (beginner) should kick

without bending the legs. The striped zebra mapping

(pattern) of the beginner’s motion is heterogeneous.

Skill Level Evaluation of Motion Curved Surface Character

501

Figure 6: Motion curved surfaces of beginner (gradient of curvature in inside kicking motion).

3.2 Correlation of Surface Character

We investigate the correlation of motion surface

characters, which are the maximum curvature and

surface area. The correlation of the maximum

curvature and the curved surface area are shown in

Figure 7. From the figure, expert’s distribution is

gathered to small regions, that is, experts repeat the

similar physical motion. On the other hand,

beginner’s distribution is large and varied (scattered).

The beginner’s maximum curvatures are large,

because the leg is bent.

Figure 7: Correlation of motion surface character.

The beginner’s surface areas are small, because

the foot motion is small. According to an expert’s

opinion, the reason is that beginner kick strongly or

weakly. Then, we decide the expert’s region, which is

0.05-0.10 rad/mm maximum curvature and 0.40-0.55

m2 surface area. If learner’s (beginner’s) data is

placed near expert’s region, the learner’s skill level is

increasing. Therefore, skill level can be evaluated

using the expert’s region.

4 KICK TRAINING USING

MOTION CURVE SURFACE

We have not confirmed the increasing level and

changing the motion curved surface characters yet,

only visualized the skill level using the characters.

Then, we investigate the character correlation using

kick training. The training method is watching a

motion movie with or without motion curved surface.

The movie is 2D viewer. The movie with motion

curved surface is to watch the expert’s curved surface

model made by ARToolKit and OpenGL library

using head mount display (Mitsuhashi et al., 2015).

First, four beginner subjects are tracked the

kicking motion in soccer. Next, two subjects watch

the movie without surface model, and the movies are

only expert’s motion. The others watch the movie

with expert’s motion curved surface model. And both

subjects train to watch or move in 10 minutes. Figure

ICINCO 2016 - 13th International Conference on Informatics in Control, Automation and Robotics

502

8 (a) shows the watching situation only the movie

without movie, Figure 8 (b) shows the training

situation with expert’s motion curved surface model,

which is placed at right lower half body (foot, knee,

and hip) side. After training, four subjects are tracked

the kicking motion again.

Figure 9 shows the before training curved surface

for four beginners. They are trapezoid (large hip

motion) or bending shape (bending the knee), or

narrow triangle shape (small foot motion). Figure 10

shows the after training curved surface. Figure 10 (a)

shows the watching movie without surface model

(Subject 1 and 2), Figure 10 (b) shows the watching

movie with surface model (Subject 3 and 4). From the

figures, all the surfaces are similar to expert’s fan-

shaped surface, and their foot trajectory is a curved

line (segment). However, motion curved surfaces of

training with the model are nearer the expert’s surface

than training without a model. We consider that train

with model is enabled to understand the lower half

body trajectory.

We investigate the skill level change from the

character correlation. Figure 11 shows the correlation

and the change of the maximum curvature and the

curved surface area. From the figure, all beginners

approach the expert’s region. The approach of

learners with surface model is larger than the learners

without model, because the learner to model move the

foot consciously. On the other hand, the learners

without model can’t recognize the lower half body

trajectory in learner’s opinion. In this result, we prove

the skill training validity using the motion curved

surface. Therefore, increasing skill level can be

evaluated using the character correlation.

5 CONCLUSIONS

We investigate the correlation of the character using

the motion curved surface, and the different and

tendency of experts and beginners is extracted. In this

result, the curved surface of expert’s motion is created

repeatedly as the same shape, and different from

many beginner surfaces. From the correlation of the

maximum curvature and the curved surface area,

expert’s distribution is gathered to small region. On

the other hand, beginner’s distribution is large and

varied. In addition, we investigate the transition of

training effectiveness for beginners from character

correlation. In this result, all the shapes of surface are

similar to expert’s shape, but motion curved surfaces

of training with the model are nearer the expert’s

surface than training without a model. According to

the character correlation, all beginners approach the

expert’s region. The approach of learners with the

surface model is larger than the learners without a

model. Therefore, we prove the skill training validity

using the motion curved surface. In future work, the

motion velocity or acceleration curved surface is

suggested, and the new method of skill training using

motion curved surface.

(a) Only watching movie (b) Motion with surface

Figure 8: Training the inside kicking motion in soccer.

(a) Only watching movie (b) Motion with surface

Figure 9: Motion curved surface of before training.

(a) Only watching movie (b) Motion with surface

Figure 10: Motion curved surface of after training.

Figure 11: Correlation of motion curved surface character

in before and after training.

ACKNOWLEDGEMENTS

This work was in part supported by JST RISTEX

Service Science, Solutions and Foundation Integrated

Research Program.

Skill Level Evaluation of Motion Curved Surface Character

503

REFERENCES

Hashimoto, H., Yoshida, I., Teramoto, Y., Tabata, H., and

Han, C., 2011. Extraction of Tacit Knowledge as Expert

Engineer's Skill based on Mixed Human Sensing,

Proceedings of the 20th IEEE International Symposium

on Robot and Human Interactive Communication,

pp.413-418, 2011.

Yoshitaka Fujino， Kousei Inoue， Masao Kikkawa，

Emi Nishina, and Tsuneo Ymamada. Sport

Onomatopoeia Data Base for Motor Learning, Japan

Society for Educational Technology, Vol.29, pp.5-8,

2005, (in Japanese)

Taki, T., Hasegawa, J. I., & Fukumura, T. (1996,

September). Development of motion analysis system

for quantitative evaluation of teamwork in soccer

games. In Image Processing, 1996. Proceedings,

International Conference on Vol. 3, pp. 815-818. IEEE.

Yasushi Takeo ， Wataru Natsu, 2011. Development of

valuation Method for Measurement Skill Training,

Proceedings of International Symposium on

Standardization Education and Research 2011 Tokyo

Japan, pp.130-145

Cheung, K. M. G., Simon Baker, and Takeo Kanade., 2003.

Shape-from-silhouette of articulated objects and its use

for human body kinematics estimation and motion

capture, Computer Vision and Pattern Recognition,

2003. Proceedings. 2003 IEEE Computer Society

Conference on. Vol. 1. IEEE, pp.77-84

Sigal, Leonid, and Michael J. Black., 2006. Humaneva:

Synchronized video and motion capture dataset for

evaluation of articulated human motion. Brown

Univertsity TR 120

Toshiyuki Murao, Yasuyuki Hirao and Hiroshi Hashimoto,

2011. Skill Level Evaluation for Taijiquan based on

Curve Fitting and Logarithmic Distribution Diagram of

Curvature, SICE Journal of Control, Measurement, and

System Integration, 4, 1, pp.001–005

Hiroshi Hashimoto, Mizuki Nakajima, Seiichi Kawata,

Sang-Gyu Shin, Toshiyuki Murao, 2014. Skill Level

Evaluation of Taijiquan based on 3D Body Motion

Analysis, Proc. Of IEEE International Conference on

Industrial Technology (ICIT2014), CD-ROM, 2014

Kaoru Mitsuhashi, Hiroshi Hashimoto, and Yasuhiro

Ohyama, 2014. The Curved Surface Visualization of

the Expert Behavior for Skill Transfer Using Microsoft

Kinect, 11th International Conference on Informatics

in Control, Automation and Robotics (ICINCO 2014),

September1-3, Wien, Austria, 2014, pp.550-555

Mitsuki Suneya, Masaki Kutsuna, Kaoru Mitsuhashi,

Yasuhiro Ohyama, and Hiroshi Hashimoto, 2014. The

Curved Surface Visualization of Behaviors for Skill

Inheriting using RGBD camera. SICE System

Integration 2014 (SI2014), December 14-17, Tokyo,

Japan. Pp.67-71, (in Japanese)

Kaoru Mitsuhashi, Hiroshi Hashimoto, and Yasuhiro

Ohyama, 2015. Motion Curved Surface Analysis and

Composite for Skill Succession using RGBD Camera,

12th International Conference on Informatics in

Control, Automation and Robotics (ICINCO 2015),

July21-23, France, 2015, pp.406-413

ICINCO 2016 - 13th International Conference on Informatics in Control, Automation and Robotics

504