Real-time Evaluation System for Top Taekwondo Athletes:
Project Overview
Pedro Cunha
1,2 a
, Paulo Barbosa
1
, Fábio Ferreira
2
, Carlos Fitas
1
, Vítor Carvalho
1,2 b
and
Filomena Soares
2c
1
2Ai - School of Technology, IPCA, Barcelos, Portugal
2
Algoritmi Research Centre, University of Minho, Guimarães, Portugal
Keywords: Deep Learning, Motion Analysis, Neural Networks, Wearable, Computer Vision, Taekwondo.
Abstract: Assessing athletes' performance is a constant challenge for coaches, whatever the sport is. In some sports
there are no technological solutions to assist coaches in this task. This is the case of Taekwondo, where
currently the methods used are mainly manual. Following this trend, this article presents the work developed
in a PhD project whose main objective is the development of a friendly and low-cost system for assessing the
performance of Taekwondo athletes in real time. Thus, the system uses a 3D camera with depth sensor
(Orbbec Astra), a computer and software developed for data collection and processing. The system also
provides the inclusion of Inertial Measurement Units (IMUs). The system allows an accurate feedback for the
correction or improvement of the athlete's techniques, enabling an increase in the athlete's performance in a
shorter period of time. In all, the project contributes to the evolution of the techniques used during Taekwondo
training, as well as to the technological development in the practice of Taekwondo.
1 INTRODUCTION
Technology plays a major role in everyday life across
the spectrum of society. Its use has been increasing,
as an integral part of the daily routine, becoming
something natural and often imperceptible. In
different areas of society and science, the use of
technology is possible through the combination of
information processing and the continuous use of
tools, extended through computing devices, known as
ubiquitous computing (Baca, Dabnichki, Heller, &
Kornfeind, 2009).
One of the areas in which research and
development of technological solutions has obtained
an active participation from the scientific community
is motion analysis. It is currently possible to analyse
the movements of humans in a natural environment
of activity, without the need for markers on the
human body. The result of these new methods of
motion analysis contributes to the creation and
availability of accessible and easy-to-use motion
capture solutions (King & Paulson, 2007).
a
https://orcid.org/0000-0001-6170-1626
b
https://orcid.org/0000-0003-4658-5844
c
https://orcid.org/0000-0002-4438-6713
In sport, the evolution in movement analysis has
allowed the development of technological solutions
that help athletes, coaches and referees in certain
tasks (Thomas, Gade, Moeslund, Carr, & Hilton,
2017). These technological solutions are divided in
two main groups. The non-optical systems, which
uses sensors placed on the athlete's body to obtain
movement information. And the optical systems, with
or without markers. The optical systems with markers
show better results combined with a higher cost and
complexity of implementation, as well as higher
intrusiveness, and systems without markers are easier
to implement, being less intrusive (Pueo & Jimenez-
Olmedo, 2017).
Assessing the performance of athletes is a
complex and difficult task in any sport. The inclusion
of motion analysis in the practice of sport, through the
systems developed, came to assist in this task. Some
of the systems developed to perform the evaluation
allows to obtain relevant information of the athlete’s
performance like velocity, acceleration, force,
displacement, among other characteristics (Arastey,
Cunha, P., Barbosa, P., Ferreira, F., Fitas, C., Carvalho, V. and Soares, F.
Real-time Evaluation System for Top Taekwondo Athletes: Project Overview.
DOI: 10.5220/0010414202090216
In Proceedings of the 14th International Joint Conference on Biomedical Engineering Systems and Technologies (BIOSTEC 2021) - Volume 1: BIODEVICES, pages 209-216
ISBN: 978-989-758-490-9
Copyright
c
2021 by SCITEPRESS Science and Technology Publications, Lda. All rights reserved
209
2020) (Cunha, Carvalho, & Soares, 2018) (Nadig &
Kumar, 2015).
In Taekwondo martial art, the evaluation athletes’
performance is still carried out by traditional
methods, that is, by viewing videos of the training
sessions or in loco of the athletes' movements in real
time. This is time consuming for the coach and delays
the feedback for improvements to the athlete (Pinto,
et al., 2017).
Figure 1: Paper flowchart.
This paper intends to present a project whose
objective is to assess the performance of Taekwondo
athletes in real time. The study performed during the
project aims to contribute with a new method of
identifying and quantifying the movements
performed by the taekwondo athlete during training
sessions using deep learning methodologies applied
to the data collected from the taekwondo athletes'
movements in real time.
This paper is organized into four chapters as
described by the flowchart presented in figure 1. The
second chapter presents the state of the art; in the third
chapter, the work and study performed on the tasks
that make up the main project are described; and in
the fourth chapter are presented the final remarks
along with the future work in the project
development.
2 STATE OF ART
Along the development of technology, several
devices have emerged that enable the monitoring and
analysis of movements performed by the human
body. Taking advantage of this, in Sports area has
been conducted a large number of researches aiming
to contribute to the improvement of athletes'
performance and help in the prevention of injuries.
For trainers and athletes, motion analysis is of
great importance when applied in the training because
it allows to provide technical skills improvement by
correcting the trainee’s body motion in order to
perform correct and most efficient movements in any
sports.
Regarding motion analysis some of the research
carried out aims to study the hands movements and
localization. As the case of the study that presents a
survey where they summarized the techniques used
for hand localization and gesture classification
(Suarez & Murphy, 2012).
Despite a growing increase in image resolution,
traditional video cameras are conditioned by the
luminosity and colours present in the environment
making it difficult to obtain the correct digital
analysis of the image. Depth cameras appear as the
suitable alternative for image collection when these
situations occur, collecting depth images, not
dependable on orientation or intensity of illumination,
or from the colour scheme of the environment. One
of the most used depth sensors in scientific research
regarding gesture classification and hand localization
is the Microsoft Kinect (Microsoft, 2020); mainly due
to its low acquisition value, portability, not
requirement of markers, ease to set up, and creation
of 3D images. It allows to develop affordable 3D
video motion systems that are used to human
movement kinematics analysis of body joints and
segments in several areas.
BIODEVICES 2021 - 14th International Conference on Biomedical Electronics and Devices
210
A Microsoft Kinect camera used to collect data
from whole human body movements was presented in
a study for human gesture recognition using data
mining classification methods in video streaming of
twenty body-joints positions of the human body
(Patsadu, Nukoolkit, & Watanapa, 2012). The study
recognized gesture patterns as stand, sit down, and lie
down. The classification methods adopted for
comparison were backpropagation neural networks,
decision trees, support vector machines and naive
Bayes. The results obtained allowed to conclude that
backpropagation neural networks show superior
performance compared to the other classification
methods, recognizing human gestures with 100%
accuracy. The average accuracy of all classification
methods was 93.72%, confirming the efficiency of
the Kinect camera when used in human body
recognition applications.
Also, with Microsoft Kinect, a study was carried
out where compared the Microsoft Kinect
displacement measures with the Peak Motus marker-
based system displacement measures. The results
allowed to conclude that the Microsoft Kinect, for
being a mark less system, was more favourable during
the setup, data collection and analysis stages as
compared to the Peak Motus (Zerpa, Lees, Patel,
Pryzsucha, & Patel, 2015).
A real-time evaluation approach that uses a
Microsoft Kinect and image processing techniques to
recognize and record the number of occurrences of a
movement in Taekwondo training environment is
presented by Pinto, et al (2017). The recognition of
the movements was obtained through the calculation
of the angles between human body joints and
comparing them with the correct values of each
movement previously saved in a database.
In (Cunha, Carvalho, & Soares, Development of a
Real-Time Evaluation System For Top Taekwondo
Athletes SPERTA, 2018) it is presented a system to
evaluate the performance of Taekwondo athletes
during training sessions. The 3D camera used is the
Orbbec Astra that comparing with Microsoft Kinect 2
is smaller, has less weight, with a higher maximum
reach distance. The system allows to save information
about the athletes and about the movements.
Regarding movements the data obtained are relative
to Cartesian coordinates in the real world of the
human body joints. Providing athlete hands and feet
joints movements data in Cartesian coordinates
numeric values, Cartesian coordinates line chart and
speed line chart, all in real-time.
Still further studies are needed, so that it can be
confirmed the reliability and validity of the Microsoft
Kinect for human movement kinematics analysis
(Polak, Kulasa, VencesBrito, Castro, & Fernandes,
2016). However, some studies verified and agreed
that marker less systems would be a major revolution
in the analysis of human motion making possible the
application of human motion capture studies
(Robertson, Caldwell, Hamill, Kamen, & Whittlesey,
2004) (Corazza, Mundermann, Gambaretto, Ferrigno,
& Andriacchi, 2010).
Other studies follow a different approach, where
movement data is collected using motion sensors used
by athletes. These wearable devices are valuable
instruments for the improvement of sports
performance. However, the existing systems are still
limited (Li, et al., 2016).
In (Camomilla, Bergamini, Fantozzi, & Vannozzi,
2018) the authors agreed that magneto-inertial
technology is a reliable tool to improve athlete’s
performance, the training specificity and to prevent
injuries. These sensors measurements can be used to
estimate temporal, dynamic and kinematic
parameters.
Smart sensors and sensor fusion allow to study the
impact suffered by the athlete. In (Mendes Jr, Vieira,
Pires, & Stevan Jr, 2016), the authors demonstrated
the use of smart sensors and sensor fusion in
biomedical applications and sports areas, promoting
a reflection about techniques and applications to
process physical variables associated with the human
body. The application can be used in areas related to
rehabilitation, the athlete’s performance
development, among others.
In (Amaro, et al., 2017), the authors agreed that
the impact signals combined with IMU may be a
reliable way of scoring, whilst heart rate
measurement enables monitoring of the athlete’s
physical state. The technique used consists in
integrating a “non-invasive” sensor system into
Taekwondo clothes. The impact is measured using
pressure sensors, thin film piezo resistive force and
accelerometers. The communication between the
sensor and the computer is based on Bluetooth and it
was discovered a limitation of bandwidth using this
transmission protocol.
3 PROJECT DEVELOPMENT
The project from which this study derives aims to
contribute with a technological solution that allows
the assessment of the performance of Taekwondo
athletes in real time during training sessions. In order
to achieve the objective, several tasks have to be
completed, each one contributing with elements
necessary for the development of the system.
Real-time Evaluation System for Top Taekwondo Athletes: Project Overview
211
Thus, the main outputs of system will be statistics,
biomechanics and motion analysis. The statistical
analysis, will be made with the results obtained
through the identification and quantification of the
movements performed by the athlete, allowing to
assess the evolution over time of the training sessions.
And the biomechanics and motion analysis, will
allow to calculate acceleration, velocity and the
applied force of the athlete’s movements.
In this chapter these tasks will be presented,
referring to their role in the functioning of the final
system.
3.1 Framework
The developed system is composed by a 3D camera
Orbbec Astra, a computer and by the software, as
presented in figure 2. The option for the 3D Camera
Orbbec Astra is due to the fact that it is smaller in size,
weighs less, does not require external power and has
a higher maximum range system, compared to
Microsoft Kinect 2, the most used depth sensor in
research (Cunha, Carvalho, & Soares, Development
of a Real-Time Evaluation System For Top
Taekwondo Athletes SPERTA, 2018).
Figure 2: Framework system architecture (Cunha,
Carvalho, & Soares, 2018).
The system allows to collect data on movements
performed by taekwondo athletes during training
sessions, calculating and presenting the values of
speed, acceleration and applied force of the athlete's
hand and feet in real time (Cunha, Carvalho, &
Soares, Development of a Real-Time Evaluation
System For Top Taekwondo Athletes SPERTA,
2018).
3.1.1 Software
The software of the framework was developed in C#
with Visual Studio 2017 IDE (Integrated
Development Environment) for build up the interface
and the main features, along with Structure Query
Language (SQL) to add the database where the data
is stored. In order to obtain athletes movements data,
the Nuitrack™ SDK was integrated. Nuitrack™ is a
3D skeleton tracking gesture recognition solution
middleware that allows to obtain the athletes joints
cartesian coordinates relative to the range of the 3D
camera Orbbec Astra.
Figure 3: Raw data from Left Ankle Joint.
The development of the software had as initial
objective to create a dataset with the data of the
movements that the athletes perform during the
practice of Taekwondo. The dataset consists of
several classes, each referring to a movement, in
which the cartesian coordinates of the athlete joints
for each class are stored. The values of each joint refer
to the X, Y, and Z coordinates in a tri-dimensional
environment.
Figure 4: Depth sensor, cartesian coordinates and speed
data output in real-time (Cunha, Carvalho, & Soares, 2018).
It is intended to integrate the software with the
functionality of identifying and quantifying the
movements performed by Taekwondo athletes during
training sessions. For that, motion analysis will be
performed according to skeleton-based action
recognition, through deep learning methodologies.
BIODEVICES 2021 - 14th International Conference on Biomedical Electronics and Devices
212
The main propose of the developed dataset is to
gather information about the taekwondo athletes’
movements aiming to use on training of deep learning
classification methods. The dataset obtained consists
of four classes (front leg bandal, rear leg bandal,
jirugui and front leg miro), in which each class
represents a movement performed by the taekwondo
athlete, with about 200 samples per class. The data
acquisition and treatment were performed to identify
and quantify the athlete techniques.
In order to select the deep learning method that
best fits the type of data, several deep learning were
studied and tested, such as, Convolutional Neural
Networks (CNN), Long Short-Term Memory
(LSTM), Graph Convolution Networks (GCN),
CNN+LSTM and a LSTM with embedded
Convolution (ConvLSTM), figure 5.
Figure 5: Deep learning methods testing system diagram.
During the tests, the classification validation
accuracy was 80% for the CNN model, 88% for the
LSTM model, 93% for the CNN+LSTM model, and
92% for the ConvLSTM model. The results obtained
allow to realize that convolutional layers models
achieve the best results. Both CNN+LSTM (figure 6)
and ConvLSTM, managed to get results above 90%
on accuracy validation, placing these models as the
ones that best fit the characteristics of the Taekwondo
athlete’s performance evaluation system.
Figure 6: CNN+LSTM Network Architecture.
3.2 Inertial Measurement Units (IMUs)
For the movements data acquisition, as presented
above, was used a system data extract the body joints
coordinates in a tri-dimensional environment.
Although this method allows data to be collected
efficiently, sometimes, due to a rotation of the athlete
or overlapping of a limb, there is occlusion. In order
to overcome these occlusions, the addition of motion
sensors, more specifically the inertial measurements
units, was foreseen. They will be positioned on the
extremities of the upper and lower limbs, hands and
feet. For this purpose, custom-made containers were
designed to accommodate the various hardware
components, which will be fixed to the athletes
through a velcro system.
As Taekwondo is a sport in which the athlete
performs fast and wide movements, with a greater
incidence of the lower limbs, the size and weight of
the motion sensors must be considered. The solution
developed should be the least intrusive as possible,
not interfering with the athlete's movements and with
adequate comfort for use during training sessions.
Along with the characteristics related to the
intrusiveness of motion sensors, the value and ease of
acquisition of the components were also considered.
Figure 7: Wemos D1 mini Wi-Fi board (a) and the GY 521
MPU 6050 (b) connection diagram.
Thus, for data processing and transmission it was
selected the Wemos D1 mini a Wi-Fi board based on
ESP-8266, with 11 digital input/output pins and 1
analogue input, as seen in figure 7 a) (LOLIN D1
mini, 2020).
The sensor chosen to obtain the acceleration and
gyroscope data was the GY 521 MPU 6050 (figure 7
b)), which is a three-axis gyroscope and acceleration
module, with standard communication I2C. The
acceleration range is between ±2 and ±16 g and the
gyroscope range is between +250 to +2000 ̊/s
(MPU6050 - Accelerometer and Gyroscope Module,
2020). The motion sensors system also includes a
battery shield (Battery Shield, 2020) which allows to
choose between powering the system through a
battery or through a USB charger, as well as enable
to charge the battery when the system is being
powered by the USB charger.
The system presented in figure 8 measures the
displacement and acceleration of the upper and lower
limbs, transmitting this data through a Wi-Fi
communication, stored in the computer belonging to
the system.
For this, a program was developed that allows the
reading of the IMUs and the transmission of data
through the UDP protocol.
Real-time Evaluation System for Top Taekwondo Athletes: Project Overview
213
Figure 8: Motion sensors system architecture diagram.
3.3 Mobile APP
Mobile devices are a constant presence in everyday
life, and some devices currently have processing
capacity comparable to computers. Therefore, they
are also used as a tool to perform various tasks.
Taking this into account, together with the
development of the framework presented above in
which the data processing fell on a portable computer
with Windows operating system, an app was
developed for Android and IOS devices.
The app is intended to be a friendly tool that can
be used by the trainer during training sessions. It will
allow to manually enter the movements performed by
the athlete, with the objective of providing a fast
feedback to the coach and athletes so that they can
analyze, correct and adapt the training method to
improve their performance.
The app makes it possible to enter athletes' data,
such as name, weight category, step and club, saving
this information in a database to add to training
sessions and consultations.
After filling in the training data, when selecting
"Start training", the menu displayed in figure 9
appears, where: in a) it is possible to select the
movements to be performed. A maximum of 10 types
of movements will be allowed per training session,
which can be leg and/or arm movements; in b) the
type of movement (arms or legs) can be selected; in
c) the duration of the training is presented; in d) the
buttons to start or end the training are available; in e)
four additional buttons are available for each
movement where the user (trainer) can specify the
type of movement: right front (1st column), right back
(2nd column), left front (3rd column), left back (4th
column) ); in f) there is a button available to indicate
whether the movement was successful.
Figure 9: App training session interface.
After the training is finished, a small summary of
the training is presented, indicating when the training
started and ended and its duration, the number and
type of movements performed during the session,
which movements were performed with or without
success.
4 FINAL REMARKS
The purpose of this paper is to present the project
under development, whose main objective is to
design and implement a friendly and low-cost system
for assessing the performance of Taekwondo athletes
in real time. The system should have the lowest level
of intrusion to athletes during the practice of
Taekwondo.
A framework for the acquisition of movement
data was developed, which allowed the creation of a
dataset with data on the movements of Taekwondo
BIODEVICES 2021 - 14th International Conference on Biomedical Electronics and Devices
214
athletes. With this dataset it was possible to conduct
a study on deep learning methodologies in order to
define which method has the best performance in
identifying athletes' movements.
With the intention of promoting the inclusion of
technological tools in Taekwondo training sessions
and taking advantage of the easy access to mobile
devices, a mobile App for Android and IOS was
developed. This App allows the trainees to add
information about the athletes and manually collect
data about the movements performed by the athlete in
a training session, saving them in the application's
database. Afterwards, the saved sessions can be
consulted, allowing to analyse the evolution of the
athlete's performance.
As future work, we intend to include the deep
learning model to identify the athlete's movements in
the developed framework and the integration of the
data acquisition system through motion sensors based
on IMUs in the framework.
Then, tests of the overall system will be
performed with athletes in a training environment, in
order to assess the impact of the system developed in
the practice of Taekwondo training and its
contribution to the assessment of the performance of
Taekwondo athletes.
ACKNOWLEDGEMENTS
This work has been supported by COMPETE: POCI-
01-0145-FEDER-007043 and by FCT – Fundação
para a Ciência e Tecnologia within the R&D Units
Project Scope: UIDB/00319/2020
. Pedro Cunha
thanks FCT for the PhD scholarship
SFRH/BD/121994/2016.
REFERENCES
Amaro, B., Antunes, J., Cunha, P., Soares, F., Carvalho, V.,
& Carvalho, H. (2017). Monitoring of Bioelectrical and
Biomechanical Signals in Taekwondo Training: First
Insights. Em European Congress on Computational
Methods in Applied Sciences and Engineering (pp. 417-
426). Springer.
Arastey, G. M. (1 de 12 de 2020). Computer Vision In
Sport. Obtido de SPORT PERFORMANCE
ANALYSIS:
https://www.sportperformanceanalysis.com/article/co
mputer-vision-in-sport
Baca, A., Dabnichki, P., Heller, M., & Kornfeind, P.
(2009). Ubiquitous computing in sports: A review and
analysis. Journal of Sports Sciences, 27, pp. 1335-1346.
Battery Shield. (28 de 11 de 2020). Obtido em 28 de 11 de
2020, de WEMOS:
https://www.wemos.cc/en/latest/d1_mini_shield/batter
y.html
Camomilla, V., Bergamini, E., Fantozzi, S., & Vannozzi,
G. (2018). Trends supporting the in-field use of
wearable inertial sensors for sport performance
evaluation: A systematic review. Sensors, 18(3), 873.
Corazza, S., Mundermann, L., Gambaretto, E., Ferrigno,
G., & Andriacchi, T. P. (2010). Markerless motion
capture through visual hull, articulated icp and subject
specific model generation. International journal of
computer vision, 87(1-2), 156-169.
Cunha, P., Carvalho, V., & Soares, F. (2018). Development
of a Real-Time Evaluation System For Top Taekwondo
Athletes SPERTA. SENSORDEVICES 2018, The Ninth
International Conference on Sensor Device
Technologies and Applications, pp. 140-145.
Cunha, P., Carvalho, V., & Soares, F. (2018). Real-Time
Data Movements Acquisition of Taekwondo Athletes:
First Insights. International Conference on Innovation,
Engineering and Entrepreneurship, pp. 251-258.
King, B. A., & Paulson, L. D. (2007). Motion capture
moves into new realms . Computer, 40(9), pp. 13-16.
Li, R. T., Kling, S. R., Salata, M. J., Cupp, S. A., Sheehan,
J., & Voos, J. E. (2016). Wearable performance devices
in sports medicine. Sports health, 8(1), 74-78.
LOLIN D1 mini. (28 de 11 de 2020). Obtido em 28 de 11 de
2020, de WEMOS:
https://www.wemos.cc/en/latest/d1/d1_mini.html
Mendes Jr, J. J., Vieira, M. E., Pires, M. B., & Stevan Jr, S.
L. (2016). Sensor Fusion and Smart Sensor in Sports
and Biomedical Applications. Sensors, 16(10), 1569.
Microsoft. (30 de 11 de 2020). Kinect para Windows.
Obtido de Microsoft:
https://developer.microsoft.com/pt-pt/windows/kinect/
MPU6050 - Accelerometer and Gyroscope Module. (28 de
11 de 2020). Obtido em 28 de 11 de 2020, de
Components101:
https://components101.com/sensors/mpu6050-module
Nadig, I. S., & Kumar, N. (2015). Measurement of velocity
and acceleration of human movement for analysis of
body dynamics. Measurement, 3(3).
Patsadu, O., Nukoolkit, C., & Watanapa, B. (2012). Human
gesture recognition using Kinect camera. (IEEE, Ed.)
ninth international conference on computer science and
software engineering (JCSSE), pp. 28-32.
Pinto, T., Faria, E., Cunha, P., Soares, F., Carvalho, V., &
Carvalho, H. (2017). Recording of occurrences through
image processing in Taekwondo training: First insights.
Lect. Notes Comput. Vis. Biomech, pp. 427-436.
Polak, E., Kulasa, J., VencesBrito, A., Castro, M. A., &
Fernandes, O. (2016). Motion analysis systems as
optimization training tools in combat sports and martial
arts. Revista de Artes Marciales Asiáticas, 10(2), 105-
123.
Pueo, B., & Jimenez-Olmedo, J. M. (2017). Application of
motion capture technology for sport performance
analysis. Retos, 32, pp. 241-247.
Real-time Evaluation System for Top Taekwondo Athletes: Project Overview
215
Robertson, D. G., Caldwell, G. E., Hamill, J., Kamen, G.,
& Whittlesey, S. (2004). Research methods in
biomechanics. USA: Human Kinetics.
Suarez, J., & Murphy, R. R. (2012). Hand gesture
recognition with depth images: A review. Em ROMAN:
The 21st IEEE International Symposium on Robot and
Human Interactive Communication (pp. 411-417).
IEEE.
Thomas, G., Gade, R., Moeslund, T. B., Carr, P., & Hilton,
A. (2017). Computer vision for sports: Current
applications and research topics . Computer Vision and
Image Understanding, 159, pp. 3-18.
Zerpa, C., Lees, C., Patel, P., Pryzsucha, E., & Patel, P.
(2015). The use of microsoft Kinect for human
movement analysis. International journal of sports
science, 5(4), 120-127.
BIODEVICES 2021 - 14th International Conference on Biomedical Electronics and Devices
216