Telekin

Tele-rehabilitation System for Musculoskeletal and Cognitive Disorders

using Natural Movement Interface Devices

Raúl Velasco Caminero

, Luis A. Méndez Herrero

, Francisco J. Díaz-Pernas

, Juan Calabia del

Campo

, Míriam Antón Rodríguez

and Mario Martínez-Zarzuela

Department of Signal Theory and Communications and Telematics Engineering, University of Valladolid, Spain

Faculty of Health Sciences, European University Miguel de Cervantes, Spain

Keywords: Rehabilitation, Serious Games, Microsoft Kinect®, Leap Motion.

Abstract: In this paper we describe a virtual rehabilitation system designed to improve different physical and cognitive

disorders. Rehabilitation is carried out through virtual reality and serious games. Currently, in this system, we

use Microsoft Kinect v2 and Leap Motion sensors. Thanks to the modular development of the system, we can

add new rehabilitation devices such as Oculus Rift, as well as create new games depending on the pathology

to be treated. Therapists access to the system through a web portal where, depending on their access level,

they can create rehabilitation sessions and therapies and track the patients’ progress. Patients access to the

system through a local application and not using keyboard or mouse devices, using a natural user interface

based on their movements. Rehabilitation sessions consist on serious games where the patient performs the

exercises of his session programmed by a therapist. There are different types of user with different access

levels: administrator, therapist, familiar and patient. The different types of user interact with their respective

modules.

1 INTRODUCTION

The increase of life expectancy among the population

brings along, in many cases, chronic diseases that

reduce their mobility and prevent them from

developing routine tasks such as getting dressed,

showering… becoming dependent people

(Christensen K, 2009). In addition, these deficits can

reduce the participation in social and leisure

activities, and even, affect in a negative way the

person’s mood (W. Gabriele, 2009). One of the

visible consequences is the saturation in clinics and

hospitals. The main factors that contribute to this

increase are the need of cares for these people

(prevention and detection of disabilities) as well as

the need of rehabilitation (Nichols-Larsen D.S.,

2005). A huge amount of physiotherapists and

professionals indicate that the number of exercises in

a therapy session is generally insufficient and the lack

of regularity of the execution of them prevent

rehabilitation from being completely effective (C.

Lang, 2007).

Researchers suggest that an occupational therapy

can stimulate the brain enough to improve patients´

motor skills. The creation and development of

motivating and effective methods to encourage these

kind of patients to perform the proposed exercises is

crucial to help them improve their motor skills and

increase their independence (J. Kleim, 2003).

An interesting strategy to follow is the

incorporation and use of various technologies focused

on occupational therapies in several environments.

Specifically, virtual reality and serious games based

on patients´ movement are gaining relevance (D.

Jack, 2000) (Beaulieu-Boire, et al., 2015) .

One of the most common problems for the

establishment of motion capture systems is their high

cost and the complexity of their assembly. However,

with the emergence of new technologies like sensors

for motion capture (Kinect v2, Intel Real Sense, Leap

Motion…), they have given rise to tools for the

objective analysis of the movement, as well as the

interaction with serious games by virtual reality

(Chang C-Y, 2012).

Several studies confirm the improvement in the

rehabilitation services of the patients by improving

their motor skills and creating an adherence to the

therapies proposed through virtual reality, using the

198

Caminero, R., Méndez-Herrero, L., Díaz-Pernas, F., Campo, J., Rodríguez, M. and Martínez-Zarzuela, M.

Telekin - Tele-rehabilitation System for Musculoskeletal and Cognitive Disorders using Natural Movement Interface Devices.

DOI: 10.5220/0006366701980203

In Proceedings of the 3rd International Conference on Information and Communication Technologies for Ageing Well and e-Health (ICT4AWE 2017), pages 198-203

ISBN: 978-989-758-251-6

sensors previously mentioned (Yao-Jen Chang,

2011).

Our research group designs and implements

virtual reality therapies based on serious games.

This paper shows the technical specification of

the rehabilitation system (TELEKIN) under

development by our research group. There are other

virtual reality systems that use Microsoft Kinect

Sensor such as KiRes (Antón D., 2013) and EPIK

(Garrote, et al., 2015), developed by our research

group, among others. Other researchers (Muñoz J.,

2013) (Pirovano M., 2012) also use Kinect based

systems in the physical rehabilitation area. The

difference between these systems previously

mentioned and TELEKIN is that, with this new

system under development, we seek to provide a tool

of support for doctors and therapists to take a more

precise and detailed control of the evolution of the

patients without having to carry out this follow-up on

a face-to-face basis while the patients perform the

rehabilitation sessions. This system is developed in a

modular way, that is to say, you can add new devices

for rehabilitation that will emerge over time.

Currently the system includes Microsoft Kinect v2

and Leap Motion sensors.

2 METHODS

2.1 Purpose of the System

The purpose of the system is to encourage patients to

carry out the training whose doctors have

recommended and keep a suitable and personalized

routine for their pathology.

Furthermore, it is intended to offer specialists

(doctors, therapists) a tool to keep track of their

patients’ progress wherever they do the exercises,

either at home or at the medical center.

2.2 Technologies

The system is divided, as will be shown later, in two

different parts: a web platform and a game.

The web platform is programmed using PHP and

jQuery languages, using Bootstrap framework (Team

Bootstrap, s.f.). For the database that feeds the web, a

MySQL language was used.

The game makes use of two devices to catch the

user’s movement:

 Microsoft Kinect v2 (Microsoft, s.f.) to catch

users’ body movements.

 Leap Motion (Leap Motion, Inc, s.f.) to catch

users’ hands movements.

The game has been programmed with the Unity

(http://unity3d.com) game engine, that allows

developing interactive content for multiplatform

systems (PC, Android, Mac, iOS, Linux, consoles).

The recommended hardware and the software

configurations needed are: Microsoft Kinect sensor

v2, Leap Motion, and a computer running Microsoft

Windows 8 or later with the Kinect and Leap Motion

drivers installed.

2.3 Main Features

The system has been programmed with a clearly

modular design, with the end to make possible to add

new modules in an easy way, with no hard changes

needed in the system parts.

The tool consists of the following modules:

Administrator Web Module: this module is a web

interface with a lateral menu that shows all the actions

that the administrator access level can perform. These

include the management of all users of the medical

center and the configuration of the entire system,

among others. The rest of the actions that this access

level performs are detailed in the following section

(see Account Types section).

Therapist Web Module: this module is similar to

the Administrator Web Module. It shows a web

interface that shows the actions allowed for this

access level (see Account Types section).

Relatives Web Module: this module shows a

summary of the patient´s main data and his evolution

in the different rehabilitation sessions. It can be

accessed by patient’s relatives, once they have the

proper access rights.

Game Module: The game module is a 3d

environment which contains rehabilitation activities

for the patients. It starts with a menu in which the user

must choose his account and play the activities or

games that the therapist has previously assigned. The

patient will be able to do the activities that were

scheduled. The aim of these activities is to achieve

rehabilitation through serious games and patient’s

range of motion measuring.

2.4 Account Types

Depending on the account type, the patient will have

access to different modules and configuration options

within one of them. Each access level grants the user

also to use functions of lower levels, in a hierarchical

manner. The highest level is administrator, followed

by doctor, family and patient.

Administrator´s Account: with an administrator´s

account, you can manage all the users of the system,

configure all the parameters of the system, assign

patients to doctors and create or modify the different

therapies for the patients.

Telekin - Tele-rehabilitation System for Musculoskeletal and Cognitive Disorders using Natural Movement Interface Devices

199

Therapist´s/medical´s Account: with this type of

account, you can create different sessions to the

patients who are in charge and observe their progress

through the visualizations of statistics obtained when

the patients perform the games configured. In

addition, you can configure the parameters of each

game and the parameters for the gamification offered

in the games and customize these parameters for each

patient.

Relative´s Account: with this type of account, you

can observe the patient´s progress. Several statistics

obtained in the realization of the sessions created for

the patient are offered. In addition, general data of

that patient are shown as the total score, level of play

and other data associated with the gamification used.

Patient account: the patient type account is only

used in the game module. It allows to access to the

sessions created previously by the doctors and to

perform the proposed activities.

2.5 Modules of the System

2.5.1 Admin Web Module

The administrator web module manages all the users

of the center (CRUD rules), as well as the

configuration of all the parameters of the system.

Through the web interface (Figure 1), all actions

associated with this level of access are performed. In

addition, this module allows to assign patients to the

allocated doctor.

Figure 1: Administrator access level web interface.

2.5.2 Therapist Web Module

In the medical/therapist web module, the creation and

configuration of the specific sessions to each patient

can be done. It is possible to configure specific

parameters for the sessions´ games, so that they are

customized for each patient. In this way, the therapies

are specifically adapted to the deficiencies of each

patient. In addition, you can observe the evolution of

each patient by displaying statistics. Initially, the user

of this module has to select the patient to treat.

Henceforth, all actions will be performed on the

selected patient.

Figure 2: Selection of the patient that be treated.

Once the patient is selected, all the actions

associated with this level of access are displayed.

Therapies and sessions are generated and configured

so that the patient can perform them (Figure 3).

Figure 3: Medical/therapist access level web interface.

2.5.3 Family Members Web Module

In the relative´s web module, you can see the progress

of the patient through statistics. Graphs are shown

with the data obtained during the different sessions

proposed. In addition, you can view the current

therapy and the most representative data of the

patient´s profile (Figure 4).

Figure 4: Family members access level web interface.

ICT4AWE 2017 - 3rd International Conference on Information and Communication Technologies for Ageing Well and e-Health

200

2.5.4 Game Module

This module is only accessible with a patient account.

The user enters the module to perform the activities

that had been scheduled by the doctor.

This part of the system is developed using Unity,

and it is a 3D environment that combine interactive

menus with 3D games with the aim of stimulate the

patient to do the exercises scheduled by his own

therapist.

The game module can be divided in three

different parts: menu, evaluation activities and

games:

Menu: once the system starts, it displays a

welcome screen with an access button. At first, the

patient must select his user and confirm that it is him

choosing a secret word which was previously

selected. Then, the system displays all the activities

that the therapist chose for this patient and wait for

the starting of the first of them.

The selection of the user (Figure 5) and the

activity (Figure 6) can be done automatically

depending on the parameters that the doctor has

chosen. To handle through the menu buttons, the

patient can use the hands to move de pointer on the

screen. The movements of the hands are tracked by

the sensors (Microsoft Kinect or Leap Motion) and if

the pointer stops over a button for a few seconds, this

button is pressed the function of said button is

executed.

Figure 5: Manual User Selection.

Figure 6: Automatic Activity Selection.

Evaluation activities: these activities are based on

goniometers which measure the patient’s joint range

of specific body joints.

On the left side of the screen a video that

represents the movement to perform is displayed,

while a message both read out and displayed at the

bottom of the screen. On the right side, an avatar

reproduces the movements of the patient.

The current angle of the patient’s joint and the

maximum value that has been reached in previous

sessions are showed at the top of the screen.

The activities can track the angles of joints of the

body (Figure 7) using the Microsoft Kinect sensor or

angles of joint of the hand (Figure 8) using Leap

Motion.

Figure 7: Body Joints Evaluation Activity.

Figure 8: Hand Joints Evaluation Activity.

Each exercise must be done for a given time,

determined by the doctor. When this time is over, the

activity is closed and the next exercise begins. This

loop continues until all the exercises that the doctor

had chosen are finished.

Games: the games are composed of two parts.

First, a user calibration is done to know the limits of

the patient when making the movement that is

necessary to play the game. When the system has

collected this data, the game begins.

The calibration (Figure 9) follows the same steps

that the evaluation games, but is focused on the

Telekin - Tele-rehabilitation System for Musculoskeletal and Cognitive Disorders using Natural Movement Interface Devices

201

specific movement of a joint. The range limits are

saved to configure the game that starts next.

Figure 9: Game Calibration.

Once the system knows the patient limits, it starts

the game which was selected by de therapists

considering the data collected in the calibration.

Two different easy to play games for

musculoskeletal disorders rehabilitation have been

developed. In one of them, (Figure 10) the patient

must move a ball vertically trying to avoid to crash

with the obstacles. In the other game, (Figure 11) the

patient has to move a platform horizontally in order

to bounce a ball and throw it against some blocks

which are on the opposite side. Ball and platform are

moved by the patient by doing flexion and extension

exercises of the joint which has been chosen by the

doctor (wrist, hip, shoulder).

Figure 10: First Musculoskeletal Disorders Game.

Finally, the system includes a game for cognitive

disorder rehabilitation. In this game, some words are

randomly displayed on the screen and the patient has

to use all of them to build a sentence (Figure 12).

There are two ways to move words: using Microsoft

Kinect or Leap Motion. The patient has to move the

pointer with his hands and put it over a word, and

when the word is selected, he must move it to a free

sentence gap, at the bottom of the screen. There are

Figure 11: Second Musculoskeletal disorders Game.

some gestures to catch the words: closing the

hand, joining the tips of index and thumb fingers or

keeping the pointer over the word for a while. The

therapist can configure how the patient should grab

the words and the difficulty of the sentences that

appear in the game.

Figure 12: Cognitive Disorders Game.

3 CONCLUSIONS

This paper presents the beta version of the TELEKIN

system. The system includes two clearly

differentiated interfaces. The web interface, where

therapists configure and evaluate the evolution of

their patients, and the serious games interface, that is

used for rehabilitation. All training sessions are

played in a virtual 3D environment.

This system has a modular design, that is, new

rehabilitation devices can be included to the system

as they arise. In addition, new modules can also be

developed depending on the diseases and needs of

each patient. Pathology treated with this system can

be both physical and cognitive. Currently, the system

includes physical rehabilitation modules for different

joints (shoulder, back, balance, coordination…). We

are now developing new modules for patients with

intellectual disability.

This system is presented as a complementary tool

for doctors and therapists. The system tracks the data

ICT4AWE 2017 - 3rd International Conference on Information and Communication Technologies for Ageing Well and e-Health

202

and the improvement of their patients and can be

accessed on a remote way in any moment they need

through the web platform of the system.

At this time, the included sensors in the system

are Microsoft Kinect v2 and Leap Motion, but thanks

to its modular structure, it is planned to add new

sensors that can measure more body joints. In the near

future, the system will include the Oculus Rift virtual

reality glasses and Gloveone haptic gloves to induce

different sensations in patients.

The aim is supply a tool to health professionals

and patients which makes the rehabilitation,

evaluation and treatment tasks easier, simpler and

much more effective.

REFERENCES

Antón D., G. A.-U. (2013). KiRes: A Kinect-based

telerehabilitation system. IEEE 15th International

Conference on e-Health Networking, Applications and

Services, 456-460.

Beaulieu-Boire, L., Belzile-Lachapelle, S., Blanchette, A.,

Desmarais, P.-O., Lamontagne-Montminy, L.,

Tremblay, C., Tousignant, M. (2015). Balance

Rehabilitation using Xbox Kinect among an Elderly

Population: A Pilot Study. Physiotherapy Journals,

Volume 5, Isue 2, 1000261, 1-5.

C. Lang, J. M. (2007). Counting repetitions: An

observational study of outpatient therapy for people

with hemiparesis post-stroke. Journal of Neurologic

Physical Therapy, págs. 3-10.

Chang C-Y, Z. M. (2012). Towards pervasive physical

rehabilitation using Microsoft Kinect. 6th Int. Conf.

Perv. Comp. tech. health, págs. 159-162.

Christensen K, D. G. (2009). Ageing populations: the

challenges ahead. Lancet, págs. 1196–1208.

D. Jack, R. B. (2000). A virtual reality-based exercise

program for stroke rehabilitation. Proceedings of the

4th international ACM conference on assistive

technologies, págs. 56–63.

Garrote, S., Herrero, A. J., Pedraza Hueso, M., González-

Gutiérrez, C., Fernández-San Román, M. V., Díaz-

Pernas, F., . . . Martínez-Zarzuela, M. (2015). EPIK,

Virtual Rehabilitation Platform Devised to Increase

Self-reliance of People with Limited Mobility.

Proceedings of the 1st International Conference on

Information and Communication Technologies for

Ageing Well and e-Health, 188-193.

J. Kleim, T. J. (2003). Motor enrichment and the induction

of plasticity before or after brain injury. Neurochemical

Research, págs. 1757–1769.

Leap Motion, Inc. (s.f.). Leap Motion setup. Obtenido de

https://www.leapmotion.com/setup/

Microsoft. (s.f.). Kinect SDK for windows. Obtenido de

https://developer.microsoft.com/en-us/windows/kinect

Muñoz J., H. O. (2013). BKI: Brain Kinect Interface, a new

hybrid BCI for rehabilitation. Games for Health, 233-

245.

Nichols-Larsen D.S., C. P. (2005). Factors influencing

stroke survivors' quality of life during subacute

recovery. Stroke, págs. 1480-1484.

Pirovano M., M. R.-B. (2012). Self-adaptive games for

rehabilitation at home. Computational Intelligence and

Games (CIG), 2012 IEEE Conference on IEEE, 179-

186.

Team Bootstrap. (s.f.). Bootstrap. Obtenido de

http://getbootstrap.com/

W. Gabriele, S. R. (2009). Work loss following stroke.

Disability and Rehabilitation, págs. 1487–1493.

Yao-Jen Chang, S.-F. C.-D. (2011). A Kinect-based system

for physical rehabilitation: A pilot study for young

adults with motor disabilities. Research in

Developmental Disabilities, págs. 2566–2570.

Telekin - Tele-rehabilitation System for Musculoskeletal and Cognitive Disorders using Natural Movement Interface Devices

203