Multimodal Cueing in Gamiﬁed Physiotherapy: A Preliminary Study

Negar Haghbin

1 a

and Marta Kersten-Oertel

1,2 b

Gina Cody School of Engineering and Computer Science, Concordia University Montreal, QC, Canada

PERFORM Centre, Concordia University Montreal, QC, Canada

Keywords:

Mobile Health Applications, Gamiﬁcation, Physiotherapy, Cueing, Multimodal Cues, Wearables.

Abstract:

Advances in mobile devices have made possible the adherence to healthy lifestyles and workout routines

with less supervision from a professional, for example, a strength trainer or physiotherapist. Mobile health

games in particular can help individuals with chronic conditions and disabilities who require physiotherapy

and rehabilitation to stay motivated and encouraged during their physiotherapy process. We developed a

mobile game application, Neblina Wrist Physio that works with a wearable motion sensor to look at the

effect of multimodal cueing, i.e. using stimuli to facilitate movement initiation and continuation, mechanisms

on wrist physiotherapy exercises. The results of our study showed that cueing in a gamiﬁed physiotherapy

environment is an effective means of keeping users entertained and engaged. At the same time, the impact of

cueing modalities in improving the quality of gamiﬁed physiotherapy exercises appears to be affected by the

difﬁculty of the exercise and the speciﬁc game.

1 INTRODUCTION

In guided ﬁtness and physiotherapy, concise instruc-

tions and feedback are needed so that patients can

both safely and effectively execute speciﬁc exer-

cises (Sch

onauer et al., 2011). Cueing is one of

the known methods of guidance in physical therapy

which can help a patient to perform activities more

effectively. Cueing has been deﬁned as using stim-

uli (either temporal or spatial) to help in the start or

continuation of a movement (Nieuwboer et al., 2007)

or simply as giving information as to what a partic-

ular exercise should look like or how it should be

carried out (Horstink et al., 1993). Although studies

have shown the efﬁcacy of physiotherapy can be im-

proved by the addition of cueing techniques (Nieuw-

boer et al., 2007), the use of an inappropriate modality

or excessive information, which may be the case when

using multiple modalities, may lead to excessive cog-

nitive workload. Combining different types of feed-

back or cueing can convey a better understanding of

a task since humans naturally interact with multiple

modalities. While multimodal cueing can keep pa-

tients informed, engaged, and even entertained, ex-

cessive information can be distracting, and confus-

ing (Baldwin et al., 2012). Thus a balance needs to

https://orcid.org/0000-0002-9598-9936

https://orcid.org/0000-0002-9492-8402

be found to determine which cues or which combina-

tion of cues can provide motivation and help a patient

to effectively perform their exercises.

Figure 1: To explore the effect of cueing in physiotherapy

we designed an application that works with a wearable sen-

sor for wrist exercises.

Physiotherapy can be a painful and tiring process

involving a series of exercises that patients must re-

peat over a long period. Unfortunately, patients often

get bored after a short period of time and lose moti-

vation making physiotherapy less effective. One so-

lution that has been widely used in recent years to en-

courage patients to do their exercises at home is to

gamify the exercise routines. Gamiﬁcation is deﬁned

as adding game elements (e.g. rewards/scoring, com-

Haghbin, N. and Kersten-Oertel, M.

Multimodal Cueing in Gamiﬁed Physiotherapy: A Preliminary Study.

DOI: 10.5220/0010396101370145

In Proceedings of the 7th International Conference on Information and Communication Technologies for Ageing Well and e-Health (ICT4AWE 2021), pages 137-145

ISBN: 978-989-758-506-7

137

petition, and levels) to a non-game context in order

to make a rather boring task like physiotherapy more

entertaining, engaging, and interesting (Marczewski,

2013). Games and mixed reality (e.g. augmented or

virtual reality) can be an ideal medium to incorporate

physiotherapy exercises with different cueing modal-

ities.

In this paper, we describe a preliminary study in-

vestigating real-time cueing (visual or auditory) in a

gamiﬁed physiotherapy environment (see Figure 1).

To explore this, we developed Neblina Wrist Physio,

an iOS mobile application consisting of two well-

known games (Breakout and Flappy Bird) that are

played by performing wrist exercises while wearing

Motsai’s Neblina

sensor.

1.1 Neblina Motion Sensor

The Neblina

core module is a low-power self-

contained Attitude and Heading Reference Sys-

tem (AHRS), Inertial Measurement Unit (IMU), and

Vertical Reference Unit (VRU) module with wire-

less connectivity developed by Motsai (see Figure 2).

It uses a Bluetooth connection to connect to smart-

phones, tablets and other devices. It also has ex-

tremely efﬁcient power management and low-energy

operation making it ideal for exploring motion in

wearable technology devices

. For our research, we

used the AHRS functionality of the Neblina

sen-

sor.

Figure 2: The Neblina

motion sensor compared to a

Canadian one-dollar coin size-wise.

2 RELATED WORK

In the following section we explore different cue-

ing modalities and gamiﬁcation in physiotherapy.

It should be noted that some researchers use the

terms cueing and feedback interchangeably. Although

strategies for feedback and cueing are similar, the

moment at which the stimulus is provided differs.

https://motsai.com/solutions/neblina/

Whereas cueing is the use of external stimuli to

facilitate movement during initiation and continua-

tion (Nieuwboer et al., 2007), feedback is used based

on a person’s performance [of a movement] during or

after an action. In the related works described below,

we choose the term that was used in the paper being

described.

2.1 Modality

Different modalities, e.g., visual, auditory, or so-

matosensory, can be used for cueing. In gen-

eral, visual cueing provides more useful informa-

tion; however, auditory information is processed more

quickly

. In movements where extra mobility is re-

quired, directing attention to visual cues might nega-

tively affect a physiotherapy process. In these cases

auditory or somatosensory cues are preferable. In

the speciﬁc context of health, the cueing and feed-

back that is normally given by a professional is vi-

sual (e.g. demonstrating the proper way to make a

move), auditory (e.g. giving verbal instruction), or

somatosensory (e.g. moving a person’s limb into

the right position or applying pressure to a speciﬁc

area of the body) (Sousa et al., 2016). The emer-

gence of computer-vision and wearable sensors is en-

abling automated cueing and feedback without the

need for constant human supervision. In these cases,

visual cueing has been deﬁned as the addition of non-

content information to visual representations and can

have various formats including graphics (e.g. arrows,

coloring, ﬂashing, etc.), on-screen narrations and/or

texts (Lin and Atkinson, 2011). The most studied

type of computer-generated auditory cueing is rhyth-

mic (Gallagher, 2017) for example the use of an un-

derlying beat in music (Ford et al., 2010) to increase

stride length and cadence in participants with Parkin-

son’s Disease.

2.2 Wearables, Cueing and

Gamiﬁcation in Physiotherapy

Wearable sensors are becoming more and more fre-

quently used in rehabilitation. Gallagher (Gallagher

et al., 2015) investigated auditory cueing, visual cue-

ing, feedback, and directed attention for cycling in

healthy adults and those with Parkinson’s Disease

(PD). The results showed that whereas for healthy

individuals only audio cueing and directed atten-

tion increases pedaling speed, for PD patients all 4

conditions affect pedaling rate positively. Another

https://www.otpbooks.com/greg-dea-feedback-and-cueing

-2/

ICT4AWE 2021 - 7th International Conference on Information and Communication Technologies for Ageing Well and e-Health

138

(a) (b)

Figure 3: (a) Extension and Flexion. (b) Ulnar and Radial deviation. Figures from (Larcombe, 2015).

study (Dowling et al., 2010) demonstrated the use

of wearable sensors in gait modiﬁcation for people

with knee osteoarthritis (KO). The authors examined

which type of real-time feedback (verbal or sensor vi-

bration feedback) is most effective in improving gait

and found that vibration feedback was better than ver-

bal. Nikmaram et al. (Nikmaram et al., 2019) did a

study on soniﬁcation (i.e., changes in musical pitch)

in post-stroke rehabilitation. They used of each indi-

vidual’s affected hand movements by capturing data

using sensors attached to the wrist and arm of a patient

to play simple tunes. Although there was no effect

of soniﬁcation on training overall, a small effect was

found on movement smoothness. Doyle et al. (Doyle

et al., 2011) used an iPhone as a motion sensor to in-

vestigate if exergaming can improve a patient’s accu-

racy in performing exercises. Participants who played

the exergame showed more accuracy over no feed-

back and limited video feedback environments. Gar-

cia et al. (Garcia and Navarro, 2014) developed a mo-

bile rehabilitation application using augmented reality

(AR) and gamiﬁcation to help individuals with ankle

sprain injuries. Their results showed that using AR

and gamiﬁcation can increase motivation and adher-

ence to rehabilitation routines. In a similar work, Shin

et al. (Shin et al., 2014) developed a task-speciﬁc in-

teractive game using virtual reality for post-stroke re-

habilitation of upper body parts. Their system showed

to be effective in improving attention, providing an

immersive ﬂow experience, and individualizing inter-

ventions during the rehabilitation process.

As seen from these related works, the context, in-

tended population, and exercise can impact the effec-

tiveness of gamiﬁcation, cueing and participant en-

gagement.

3 WRIST PHYSIOTHERAPY

EXERCISES

Wrist injuries or wrist pains are very common in peo-

ple who do sports or who do repetitive wrist motions

(e.g. typing on a keyboard or working on an assem-

bly line). In addition, wrist pain might be a result of

sudden impacts or certain diseases such as rheuma-

toid arthritis, which mostly affects the elderly. Physi-

cal therapy is a common treatment for wrist disorders.

Moreover, performing simple stretching and strength-

ening exercises regularly or as warm-ups before doing

sports can help to prevent wrist injuries

Two of the most common sets of wrist physiother-

apy exercises are (1) extension and ﬂexion and (2)

ulnar and radial deviations. Extension is deﬁned as

raising the back of the hand and ﬂexion describes the

bending of the hand down so that the palm faces in

toward the arm (Figure 3a). Ulnar deviation is the

movement of bending the wrist towards the little ﬁn-

ger’s side. Radial deviation is the movement of bend-

ing the wrist towards the thumb’s side (Figure 3b).

Normal values for wrist range of motions are 71 de-

grees of extension, 73 degrees of ﬂexion, 33 degrees

of ulnar deviation, and 19 degrees of radial devia-

tion (Kim et al., 2014).

4 USER STUDY

To study the impact of cueing modality preferences in

gamiﬁed physiotherapy, we developed a wrist therapy

application with different cueing modalities: visual,

auditory and visual and auditory cues combined. The

study involved participants playing two video games

on an iPad while having the Neblina

sensor placed

on top of their hand as can be seen in Figure 4.

https://medlineplus.gov/wristinjuriesanddisorders.html

Multimodal Cueing in Gamiﬁed Physiotherapy: A Preliminary Study

139

Figure 4: Setup of the user experiment.

Speciﬁcally, we had the following research ques-

tions:Which cueing modalities (no cueing, audio, vi-

sual, or a combination of audio and visual) are best

to improve physiotherapy exercise quality in a gami-

ﬁed environment? and Do users have fun and become

more engaged in physiotherapy exercises when they

are gamiﬁed?

4.1 Neblina Wrist Physio App

To answer our research questions, we developed an

iOS wrist therapy application consisting of two video

games: Breakout and Flappy Bird.

4.1.1 Breakout Game

Breakout is based on a popular arcade game (Fig-

ure 5a). In this game, the user controls a paddle at

the bottom of the screen to keep a ball from falling

while popping blocks at the top of the screen using

the same ball. In our version of the game, the paddle

is controlled by Neblina

sensor which is placed on

top of the user’s hand. Thus, the user has to do ulnar

and radial deviations in order to play the game.

In the Breakout game the visual cue was given

by arrows that pointed to the left or right indicating

where the paddle needed to be moved to hit the ball.

For auditory cueing, a voice saying ”Move left” or

”Move right” speaks while the ball moves downward

(a) (b)

Figure 5: Screenshots of the Breakout game. (a) The Break-

out game in Neblina Wrist Physio app. (b) The Flappy Bird

game in Neblina Wrist Physio app.

indicating where the paddle needs to be moved. We

also studied the combination of both cues.

4.1.2 Flappy Bird

Flappy Bird is a game, where traditionally, the user

controls the bird by tapping on the screen and avoid-

ing pipes appearing in the way (Figure 5b). In our

version of the game, the bird is controlled by the ex-

tension and ﬂexion of the user’s wrist. In this game,

extension of the wrist makes the bird ﬂy higher and

ﬂexion of the wrist makes the bird ﬂy lower. For this

game, visual cueing was given using arrows pointing

up or down while the bird is approaching a pipe and

audio cueing was done by having a voice say ”Move

up” or ”Move down” while the bird is approaching a

pipe. The combination of cues was also studied.

4.2 Experimental Design

The setup for the user experiment required the user to

place their right arm comfortably on a desk. A prelim-

inary study showed that participants playing Breakout

would tend to move their whole arm rather than just

the wrist, therefore, two horizontal poles were taped

to the desk to limit the movement of the user’s arm. To

perform the exercises correctly, the user’s arm should

stay still on a ﬂat surface, thus the participants were

instructed to leave their arm ﬂat on the table and only

use their wrist to move within the games. The par-

ticipant’s wrist was not placed on the desk so that it

could move freely without any restrictions (Figure 4).

In addition, for both games, the user can perform the

exercises while making a ﬁst or having their hand ﬂat.

For both games, the Neblina

sensor was placed

on top of the user’s hand and the speed of the game

and sensitivity of the sensor were calibrated for each

participant. Prior to the study users were ﬁrst in-

structed on how to do the exercises and then asked to

do each exercise without the system for 30 seconds.

Each participant played each game for 2 minutes un-

der four conditions: (1) no cue, (2) visual cue, (3) au-

ditory cue and (4) visual and auditory cue combined.

Thus each participant played 8 minutes of each game.

The ordering of which game was played ﬁrst, Break-

out or Flappy Bird was alternated. Participants were

able to rest before each round of the game and were

told they should stop at anytime if they experienced

any discomfort.

4.2.1 Pre-test Questionnaire

Prior to playing the game participants signed a con-

sent form and ﬁlled out a pre-test questionnaire con-

taining questions about participants’ experience with

ICT4AWE 2021 - 7th International Conference on Information and Communication Technologies for Ageing Well and e-Health

140

wearable sensors, video games, and guided exercis-

ing. Example questions from the questionnaire in-

clude: Have you done guided exercises or rehabili-

tation before? How often do you play video games?

Do you have any experience working with wear-

able devices containing motion sensors (e.g. HMD,

Smartwatch, Smart clothing etc.)? and Do you have

any experience doing ”gamiﬁed” ﬁtness activities

(e.g. Wii Sports, Kinect Sport Games, Zombie Run,

Ingress, etc.)? The full questionnaire was created

using Google Forms and can be accessed at: https:

//forms.gle/uGLAxVFxrkkkY8gX7.

4.2.2 Post-test Questionnaire

After ﬁnishing the experiment, participants were

asked to ﬁll out a post-test questionnaire. In the post-

study questionnaire we targeted participants’ pref-

erence and feeling regarding cueing modalities for

each game, and their perception of wearable sen-

sors and the use of games for physiotherapy. In ad-

dition, they were asked about any further sugges-

tions for improving the application. The post-test

questionnaire can be accessed at https://forms.gle/

J5xYSUZXeL5yZVyUA

5 RESULTS: USER STUDY

For the study, 10 healthy participants (4 females and

6 males, age 24 to 44) were recruited

. None of them

had any major hearing, vision or wrist problems.

5.1 Participants’ Background

According to the pre-test questionnaire, half (50%) of

the participants love new technologies and try using

them and 20% of them like new technologies but are

not eager to use them. The rest of the participants

might like or dislike them depending on the context

(see Figure 6).

Regarding the experience with guided exercises or

physiotherapy of any kind, 70% of the participants

did not have any experience. Twenty percent of the

participants had experience with conventional guid-

ance methods, for example with paper instructions

(e.g. sheet with physiotherapy exercises), a profes-

sional demonstrating exercises (e.g. in an aerobics

class), and video instructions (e.g. a yoga YouTube

Due to the COVID-19 pandemic and partial lock-down

in Montreal at the time, we could only run subjects from

within our bubble, thus limiting the study size and our abil-

ity to recruit subjects currently doing arm/wrist physiother-

apy exercises.

Figure 6: Users’ feeling towards new technologies.

video). Lastly, another twenty percent had experi-

ence using guidance with movement trackers or sen-

sors. None of our participants had experience in doing

guided exercises with augmented or virtual reality.

The pre-test questionnaire also showed that the

general perception of gamiﬁed ﬁtness activities was

positive. From the 30% of the participants who had

experience with them, 66.6% believed that they can-

not be effective in improving general health but they

are motivating. All of the participants believed that

they are fun, engaging and interesting. Participants

generally disagreed that games are frustrating, stress-

ful or discouraging. On the contrary, subjects who had

experience with team gamiﬁed ﬁtness activities (20%)

believed that doing ﬁtness activities with a team in a

gamiﬁed way can improve ﬁtness and general health.

Regarding the use of video games, almost half

of the participants (40%) rarely played video games.

Twenty percent of them played video games more

than 10 hours weekly and 30% of the participants

played video games less than 3 hours weekly. 10% of

the participants played 3 to 10 hours weekly. The ma-

jority of the participants (80%) used mobile devices

for playing games. Almost half, 40% use a desktop

computer and/or laptop to play and 10% of the par-

ticipants use a video game console. With respect to

the participants’ general feeling during game playing,

70% of them felt entertained, 40% of them felt ex-

cited and another 40% felt relaxed. These differences

of feelings could be due to differences in the type

of game play (e.g. Tetris/Sudoku type games versus

ﬁrst-person shooter games).

In terms of wearables, we found that 70% of our

participants have previously used a wearable device

containing motion sensors (e.g. HMD, smartwatch,

smart clothing) and 80% of them had experience with

motion-sensing devices (e.g. Kinect and Wii).

Multimodal Cueing in Gamiﬁed Physiotherapy: A Preliminary Study

141

Figure 7: Users’ feeling while playing Neblina Wrist Physio games.

5.2 Wearable Sensor

The post-test questionnaire queried users’ perception

of the Neblina

sensor. Participants were asked to

rate the precision on a scale of 1 to 5, ﬁve being very

precise. We found that 90% of the participants rated

the precision of the sensor to be greater than 3. The

majority of participants (70%) believed that the sen-

sor was easy to use and all of them felt comfortable

using the sensor.

5.3 Perception of Gamiﬁed

Physiotherapy

Generally, the participants favoured the idea of using

a game for doing physiotherapy exercises. They all

felt motivated and 90% of them felt that the study was

fun, engaging, and effective; however, almost half of

the participants (40%) felt the overall system was dif-

ﬁcult to work with. This was mainly due to the fact

that the sensor had to be attached to the wrist and cal-

ibrated, and thus could be alleviated in future studies

with, for example, the use of a wearable bracelet. At

the same time, all of the participants responded that

they would use this type of system for physiotherapy

or rehabilitation.

Regarding subjects’ feelings during playing the

Breakout game, albeit they mostly had positive feel-

ings such as entertained (50%), excited (40%), happy

(10%), had fun (10%), and were engaged (10%), 30%

of the participants felt stressed and one subject felt

bored. This result is more or less the same for the

Flappy Bird game with one major difference, that no-

body felt stressed during this game while 40% of the

participants felt relaxed during this game. In addition,

fewer participants felt excited and more felt bored.

These results could have been expected since during

the user study, the Breakout game was perceived to

be more challenging for the participants in compari-

son to the Flappy Bird game. When looking at the

participants’ general perception of video games, the

same participants that felt excited or stressed had an-

swered in the pre-test questionnaire that games typ-

ically make them stressed or excited. Thus, these

participants had the same perception of physiother-

apy games as typical games played for leisure. Those

participants who play games more often felt less en-

tertained and more bored during the study likely since

the main goal of these games is performing physical

therapy exercises rather than entertaining users (Fig-

ure 7). Furthermore, based on the post-study ques-

tionnaire, less excitement for a game might result in

boredom during the game (Figure 7).

5.4 Exercise Quality

Although we did not directly measure the quality of

the wrist exercises we did observe participants. Dur-

ing the game, even though participants were asked to

not move their arm and only use their wrists for the

exercises in some cases of the Breakout game, they

tended to shift their arm or rotate their wrists instead

of doing ulnar and radial deviation in order to avoid

losing. This might be directly connected to their level

of excitement or stress while playing the game. We

posit that the more engaged users become the more

likely they are to focus on gaining points and doing

well in the game than doing the exercise properly.

Thus games need to be carefully designed in order to

allow for proper movements and avoid more injury.

One way to do this is to use games where the score is

based on exercise quality.

5.5 Cueing

We queried the users on their preference about cueing

modality in terms of engagement, helpfulness in per-

formance, and distraction. Overall, the favourite cue

for Flappy Bird game was the auditory cue (40%), in

comparison to no cueing for the Breakout game (Fig-

ICT4AWE 2021 - 7th International Conference on Information and Communication Technologies for Ageing Well and e-Health

142

(a) (b)

Figure 8: Users’ preference regarding the type of cue. (a) For the Breakout game. One participant preferred the combination

of audio and visual, provided that they are timed better. (b) For the Flappy Bird game.

(a) (b)

Figure 9: Users’ opinion of types of cueing. (a) For the Breakout game. (b) For the Flappy Bird game.

ure 8). A T-test showed that there were no signiﬁcant

differences between the two games in terms of users’

opinion of cueing modalities (visual (p=0.4263), au-

dio (p=0.3375), audio + visual (p=0.4766)). We sum-

marize the results of the main ﬁndings in Figure 9.

5.5.1 Visual Cueing

For the Breakout game the majority of participants

felt that visual cues were not distracting (70%) or an-

noying (80%). In addition, half of the participants

sensed an improvement in their performance, and

40% of them felt it was helpful. For the Flappy Bird

game, fewer participants (40%) felt the visual cue had

a role in improving their performance but more sub-

jects (50%) sensed it was helpful. Almost all of the

participants agreed that the visual cues were not dis-

tracting (90%) or annoying (80%) for the Flappy Bird

game. The major difference regarding visual cues be-

tween the games was the level of engagement; 30% of

the participants thought that the visual cues had made

the Breakout game more engaging, whereas this re-

sult is 50% for the Flappy Bird game. This difference

is also likely due to the difference of difﬁculty in the

game. One participant commented that the Breakout

game was more difﬁcult so the cues helped, whereas,

for the Flappy Bird, which is more simple and easy to

play, the cue had no impact and therefore was ignored.

5.5.2 Audio Cueing

For both games over a third of the participants (40%)

felt that auditory cueing was annoying. This might be

as a result of the fact that participants were focused

on the visual elements of the games and having audio

cues repeating (“up/down” or “left/right”) became an-

noying. At the same time, by some participants the

cues were deemed helpful and were felt to improve

users’ performances. Thus it seems they did not neg-

atively impact the users’ game performance. For both

the Breakout game and Flappy Bird game, half of the

participants felt an improvement in their performance

comparing to visual cueing rounds. Half of the partic-

ipants felt audio cueing was helpful for the Breakout

game and this number increased for the Flappy Bird

game to 70% which is also higher than the visual cue-

ing. The majority of the participants agreed that au-

dio cues were not distracting when playing the Flappy

Multimodal Cueing in Gamiﬁed Physiotherapy: A Preliminary Study

143

Bird game (80%) or the Breakout game (70%). A ma-

jor difference regarding the level of engagement with

audio cues between the games was apparent. In terms

of engagement, 40% of the participants felt that audio

cues had made the Breakout game more engaging and

70% felt that for the Flappy Bird game.

5.5.3 Multimodal Cueing: Visual and Audio

In terms of the users’ perception of both visual and

audio cueing, while the combination resulted in the

same perception for helpfulness (50%) and distraction

(20%) for both of the games, the Flappy Bird game

with both modalities was considered more engaging

(50%) and more effective regarding improvement in

performance (50%) compared to the Breakout game

with both of the modalities. In addition, the Flappy

Bird game was found to be less annoying (20%) than

the Breakout game (30%).

6 DISCUSSION

The results of our work suggest that people believe

gamiﬁed physiotherapy is engaging and can help in

exercises. However, the design of these types of

games requires its own set of considerations and a

game’s level of difﬁculty needs to be customized

based on each individual’s capabilities. Furthermore,

consideration into using cueing for better timing and

smoothness of the exercise should be considered in

future work (Benoit et al., 2014). The impact of cue-

ing on cognitive load needs to be customized as well.

As one of the participants mentioned “... I ignore

them [visual cues for the Breakout game] because it

takes longer for me to process visual cues than play

based on my assumptions...”. This reﬂects some of

the previous studies that show that visual cues may

take too long to process and should not be used when

visual attention should be paid elsewhere.

Although our population believed that more chal-

lenging games might be more exciting and engaging,

this may not be the case with an elderly population.

With more engaging games people might pay more

attention to scoring rather than correctly performing

the physiotherapy exercises. Thus, for more engag-

ing games a possibility for future work would be to

include pose estimation, and use of feedback to show

a user when an exercise is not being performed cor-

rectly (Christiansen et al., 2013; Ayman Hassan et al.,

2020; de Araujo et al., 2017). Gaming mechanisms

where users lose points for not performing exercises

correctly could also be considered but must be bal-

anced with the motivation of users.

In terms of limitations, our user study was con-

ducted during the COVID-19 pandemic. Since this

study required participants to place a wearable sensor

on their hands, it limited us in terms of ﬁnding more

participants from diverse backgrounds, or those who

were currently doing rehabilitation exercises. Our

study also only included two cueing modalities (i.e.

visual cueing and auditory cueing). The next step

of this work will be to not only explore the effect of

haptic cueing and the combination of haptic with au-

ditory and visual cueing but also the effect of feed-

back modalities on basic and more challenging physi-

cal therapy exercises. Moreover, we will use in-game

scores as a quantitative measure for assessing users’

performance.

7 CONCLUSIONS

There is considerable research potential regarding

providing guidance in a virtual environment for phys-

iotherapy exercises. In addition, the emergence of

computer vision methods and wearable motion track-

ing devices has provided various ways of collecting

users’ movement data. Still, several concerns remain

that need to be addressed regarding the clinical ef-

fects of cueing and other forms of guidance on gam-

iﬁed physiotherapy movements. Most prominently,

we currently lack understanding of the connection be-

tween in-game performance and cueing and whether

this in turn impacts enjoyment and ultimately adher-

ence to physiotherapy routines. Although our study

shows promising results regarding the use of cue-

ing with basic gamiﬁed physiotherapy exercises and

a healthy population, it is currently unclear whether

our method will hold true for a more complex physio-

therapy routine and for participants who present with

motor deﬁcits and are going through an actual physio-

therapy process. In addition, since a number of phys-

ical therapy exercises need the patient to be more ac-

tive and move more often during an exercise, the use

of other cueing modalities such as haptics might be

more beneﬁcial than visual cues. Moreover, provid-

ing feedback to show if the user is performing exer-

cises correctly can continuously improve the users’

performance as was demonstrated in previous stud-

ies (Ribeiro et al., 2011; Christiansen et al., 2013).

Future investigations should examine whether having

feedback along with cueing can have the same or even

a stronger effect on users’ adherence to routines and

quality of movements and exercises.

ICT4AWE 2021 - 7th International Conference on Information and Communication Technologies for Ageing Well and e-Health

144

REFERENCES

Ayman Hassan, H., Hussein Abdallah, B., Ahmed Ab-

dallah, A., Osama Abdel-Aal, R., Reda Numan, R.,

Khaled Darwish, A., and El-Behaidy, W. (2020). Au-

tomatic feedback for physiotherapy exercises based

on PoseNet. Informatics Bulletin, Faculty of Com-

puters and Artiﬁcial Intelligence, Helwan University,

2(2):10–14.

Baldwin, C. L., Spence, C., Bliss, J. P., Brill, J. C., Wogal-

ter, M. S., Mayhorn, C. B., and Ferris, T. K. (2012).

Multimodal cueing: The relative beneﬁts of the audi-

tory, visual, and tactile channels in complex environ-

ments. Proceedings of the Human Factors and Er-

gonomics Society Annual Meeting, 56(1):1431–1435.

Benoit, C., Dalla Bella, S., Farrugia, N., Obrig, H., Mainka,

S., and Kotz, S. A. (2014). Musically cued gait-

training improves both perceptual and motor timing

in parkinson’s disease. Frontiers in Human Neuro-

science, 8.

Christiansen, R., Contreras-Vidal, J. L., Gillespie, R. B.,

Shewokis, P. A., and O’Malley, M. K. (2013). Vibro-

tactile feedback of pose error enhances myoelectric

control of a prosthetic hand. In 2013 World Haptics

Conference (WHC), pages 531–536.

de Araujo, F. X., Scholl Schell, M., and Ribeiro, D. C.

(2017). Effectiveness of physiotherapy interventions

plus extrinsic feedback for neck disorders: A system-

atic review with meta-analysis. Musculoskeletal Sci-

ence and Practice, 29:132 – 143.

Dowling, A. V., Fisher, D. S., and Andriacchi, T. P. (2010).

Gait modiﬁcation via verbal instruction and an active

feedback system to reduce peak knee adduction mo-

ment. Journal of Biomechanical Engineering, 132(7).

Doyle, J., Kelly, D., Patterson, M., and Caulﬁeld, B. (2011).

The effects of visual feedback in therapeutic exergam-

ing on motor task accuracy. In 2011 International

Conference on Virtual Rehabilitation, pages 1–5.

Ford, M. P., Malone, L. A., Nyikos, I., Yelisetty, R., and

Bickel, C. S. (2010). Gait training with progressive

external auditory cueing in persons with parkinson’s

disease. Archives of Physical Medicine and Rehabili-

tation, 91(8):1255–1261.

Gallagher, R. (2017). Cueing, feedback and directed atten-

tion embedded in a virtual environment modulate tem-

poral and spatial bicycling features of healthy older

adults and people with Parkinson’s disease. PhD the-

sis, Rutgers University - School of Health Professions.

Gallagher, R., Werner, W. G., Damodaran, H., and Deutsch,

J. E. (2015). Inﬂuence of cueing, feedback and di-

rected attention on cycling in a virtual environment:

Preliminary ﬁndings in healthy adults and persons

with parkinson’s disease. In 2015 International Con-

ference on Virtual Rehabilitation (ICVR), pages 11–

17.

Garcia, J. A. and Navarro, K. F. (2014). The mobile

rehapp

: an ar-based mobile game for ankle sprain

rehabilitation. In 2014 IEEE 3nd International Con-

ference on Serious Games and Applications for Health

(SeGAH), pages 1–6.

Horstink, M., De Swart, B., Wolters, E. C., and Berger, H. J.

(1993). Paradoxical behavior in Parkinson’s disease.

In Proceedings of the European Congress on Mental

Dysfunction in Parkinson’s Disease.

Kim, T. S., Park, D. D. H., Lee, Y. B., Han, D. G., Shim, J.,

Lee, Y. J., and Kim, P. C. W. (2014). A study on the

measurement of wrist motion range using the iphone

4 gyroscope application. Annals of Plastic Surgery,

73(2):215–218.

Larcombe, B. (2015). Improve Your Forehand Loop by Re-

laxing the Wrist.

Lin, L. and Atkinson, R. K. (2011). Using animations and

visual cueing to support learning of scientiﬁc concepts

and processes. Computers & Education, 56(3):650 –

658.

Marczewski, A. (2013). Gamiﬁcation: A Simple Introduc-

tion. Andrzej Marczewski.

Nieuwboer, A., Kwakkel, G., Rochester, L., Jones, D., van

Wegen, E., Willems, A. M., Chavret, F., Hethering-

ton, V., Baker, K., and Lim, I. (2007). Cueing training

in the home improves gait-related mobility in parkin-

son’s disease: the rescue trial. Journal of Neurology,

Neurosurgery & Psychiatry, 78(2):134–140.

Nikmaram, N., Scholz, D. S., Großbach, M., Schmidt,

S. B., Spogis, J., Belardinelli, P., M

uller-Dahlhaus,

F., Remy, J., Ziemann, U., Rollnik, J. D., and Al-

tenm

uller, E. (2019). Musical soniﬁcation of arm

movements in stroke rehabilitation yields limited ben-

eﬁts. Frontiers in Neuroscience, 13:1378.

Ribeiro, D. C., Sole, G., Haxby Abbott, J., and Milosavlje-

vic, S. (2011). A rationale for the provision of extrin-

sic feedback towards management of low back pain.

Manual Therapy, 16(3):301 – 305.

Sch

onauer, C., Pintaric, T., and Kaufmann, H. (2011). Full

body interaction for serious games in motor rehabil-

itation. In Proceedings of the 2nd Augmented Hu-

man International Conference, AH ’11. Association

for Computing Machinery.

Shin, J., Ryu, H., and Jang, S. H. (2014). A task-speciﬁc in-

teractive game-based virtual reality rehabilitation sys-

tem for patients with stroke: a usability test and two

clinical experiments. Journal of NeuroEngineering

and Rehabilitation, 11(1):32.

Sousa, M., Vieira, J., Medeiros, D., Arsenio, A., and Jorge,

J. (2016). Sleevear: Augmented reality for rehabil-

itation using realtime feedback. In Proceedings of

the 21st International Conference on Intelligent User

Interfaces, IUI ’16, page 175–185. Association for

Computing Machinery.

Multimodal Cueing in Gamiﬁed Physiotherapy: A Preliminary Study

145