Augmented Human Assistance (AHA)

Élvio Rúbio Gouveia

, Alexandre Bernardino

, Mónica S. Cameirão

John Muñoz Cardona

, Afonso Gonçalves

, Teresa Paulino

, Honorato Sousa

João Avelino

, Hugo Simão

, Maria de Fátima Marcelina Baptista

Maria Filomena Araújo da Costa Cruz Carnide

, David Belo

4,7

, Hugo Gamboa

4,7

Assim Smailagic

, Dan Siewiorek

, Min Hun Lee

, Martina Čaić

, Dominik Mahr

Gaby Odekerken-Schröder

, and Sergi Bermúdez i Badia

LARSYS, Madeira-ITI, University of Madeira, Portugal

LARSyS, Instituto de Sistemas e Robótica, Instituto Superior Técnico,

University of Lisbon, Portugal

CIPER, Faculdade de Motricidade Humana, Universidade de Lisboa, Portugal

PLUX - Wireless Biosignals S.A, Portugal

Carnegie Mellon University, U.S.A.

School of Business and Economics, Department of Marketing & Supply Chain Management,

Maastricht University, The Netherlands

LIBPhys-UNL, FCT, Universidade Nova de Lisboa, Portugal

Abstract. Aging and sedentarism are two main challenges for social and health

systems in modern societies. To face these challenges a new generation of ICT

based solutions is being developed to promote active aging, prevent sedentarism

and find new tools to support the large populations of patients that suffer chronic

conditions as result of aging. Such solutions have the potential to transform

healthcare by optimizing resource allocation, reducing costs, improving diagno-

ses and enabling novel therapies, thus increasing quality of life.

The primary goal of the “AHA: Augmented Human Assistance” project is to de-

velop novel assistive technologies to promote exercise among the elderly and

patients of motor disabilities. For exercise programs to be effective, it is essential

that users and patients comply with the prescribed schedule and perform the ex-

ercises following established protocols. Until now this has been achieved by hu-

man monitoring in rehabilitation and therapy session, where the clinicians or

therapists permanently accompany users or patient. In many cases, exercises are

prescribed for home performance, in which case it is not possible to validate their

execution. In this context, the AHA project is an integrative and cross-discipli-

nary approach of 4 Portuguese universities, the CMU, and 2 Portuguese industry

partners, that combines innovation and fundamental research in the areas of hu-

man-computer interaction, robotics, serious games and physiological computing

(see partner list in Appendix A). In the project, we capitalize on recent innova-

tions and aim at enriching the capabilities and range of application of assistive

devices via the combination of (1) assistive robotics; (2) technologies that use

well-understood motivational techniques to induce people to do their exercises in

the first place, and to do them correctly and completely; (3) tailored and relevant

guidance in regard to health care and social support and activities; and (4) tech-

nologies to self-monitoring and sharing of progress with health-care providers,

Gouveia, ÃL’., Bernardino, A., CameirÃ

co, M., Cardona, J., GonÃ

galves, A., Paulino, T., Sousa, H., Avelino, J., SimÃ

co, H., Baptista, M., Carnide, M., Belo, D., Gamboa, H., Smailagic, A.,

Siewiorek, D., Lee, M., Ä

NaiÄ

G, M., Mahr, D., Odekerken-SchrÃ˝uder, G. and BermÃždez i Badia, S.

Augmented Human Assistance (AHA) - Active aging Â˚u Serious games Â˚u Assistive robotics Â˚u Augmented reality Â˚u Virtual coach.

DOI: 10.5220/0008861800200041

In OPPORTUNITIES AND CHALLENGES for European Projects (EPS Portugal 2017/2018 2017), pages 20-41

ISBN: 978-989-758-361-2

enabling clinicians to fine-tune the exercise regimen to suit the participant’s ac-

tual progress.

We highlight the development of a set of exergames (serious games controlled

by the movement of the user’s body limbs) specifically designed for the needs of

the target population according to best practices in sports and human kinetics

sciences. The games can be adapted to the limitations of the users (e.g. to play in

a sitting position) so a large fraction of the population can benefit from them. The

games can be executed with biofeedback provided from wearable sensors, to pro-

duce more controlled exercise benefits. The games can be played in multi-user

settings, either in cooperative or competitive mode, to promote the social rela-

tions among players. The games contain regional motives to trigger memories

from the past and other gamification techniques that keep the users involved in

the exercise program. The games are projected in the environment through aug-

mented reality techniques that create a more immersive and engaging experience

than conventional displays. Virtual coach techniques are able to monitor the cor-

rectness of the exercise and provide immediate guidance to the user, as well as

providing reports for therapists. A socially assistive robot can play the role of the

coach and provide an additional socio-cognitive dimension to the experience to

complement the role of the therapist. A web service that records the users’ per-

formances and allows the authorized therapists to access and configure the exer-

cise program provides a valuable management tool for caregivers and clinical

staff. It can also provide a social network for players, increasing adherence to the

therapies.

We have performed several end-user studies that validate the proposed ap-

proaches. Together, or in isolation, these solutions provide users, caregivers,

health professionals and institutions, valuable tools for health promotion, disease

monitoring and prevention.

Keywords: Active aging · Serious games · Assistive robotics

Augmented reality · Virtual coach

1 Introduction

The elderly population is increasing worldwide and in combination with sedentarism

and a longer life expectancy represents one of the most important challenges our soci-

eties will face in the near future (WHO, 2012). Just the 'Health, demographic change

and well-being' theme alone in the Horizon 2020 European Framework has an indica-

tive budget of €8 billion associated European research and innovation work. Scientific

evidence based on numerous studies show that elderly who are physically active have

lower rates of chronic conditions such as cardiovascular diseases, diabetes, cancer, hy-

pertension and obesity (WHO, 2012). Chronic patients experience an increased loss of

independence, autonomy and low self-esteem, and consequently require of continuous

rehabilitation and therapy. There is the need, therefore, not only to develop strategies

to support healthy habits, prevent sedentarism and promote active aging, but to also

support those with age related chronic conditions.

Research interest in service robotics for active aging and health care has grown in

Augmented Human Assistance (AHA) - Active aging Â˚u Serious games Â˚u Assistive robotics Â˚u Augmented reality Â˚u Virtual coach

the last few decades with potential applications on healthy, elderly or patients. The ef-

fectiveness of the use of robotic devices in elderly care (Bemelmans et al., 2012;

Broekens et al., 2009), rehabilitation (Matarić et al., 2007), autism diagnosis and ther-

apy (Scassellati, 2007) and weight loss applications (Kidd & Cynthia 2008), for exam-

ple, have been empirically demonstrated in a number of works and raise great expecta-

tions on the use of robots as personal assistants. The Nursebot platform demonstrated

the ability to contact a resident, remind them of an appointment, accompany them to

that appointment, as well as provide information of interest to that person (Pineau et

al., 2003). In touch Health deployed their robot in a Neurology Intensive Care Unit and

a study suggested improvement in critical care nursing team satisfaction (Rincon et al.,

2012). The Kompaï R&D robot has been tested for elderly assistance using a diary

application for monitoring the medication and give information about daily events (Rin-

con et al., 2012). The autom robot is a weight-loss social robot (Kidd et al., 2012) that

asks you about what you've eaten and how much you've exercised, proving helpful

suggestions and feedback that's different every day and customized just for you, utiliz-

ing facial expressions and a simple touchscreen interface. In this context, the advances

in information, robotic and assistive technologies have the potential to increase quality

of life and change health care delivery models, reducing costs, and improving monitor-

ization. The “AHA: Augmented Human Assistance” project is a novel, integrative and

cross-disciplinar approach of 4 portuguese universities, CMU and 2 portuguese indus-

try partners that combines innovation and fundamental research in the areas of human

computer interaction, robotics, serious games and physiological computing. AHA’s

goal is to develop a new generation of ICT based solutions that have the potential to

transform healthcare by optimizing resource allocation, reducing costs, improving di-

agnoses and enabling novel therapies, thus increasing quality of life. The project pro-

poses the development and deployment of a novel Robotic Assistance Platform de-

signed to support healthy lifestyle, sustain active aging, and support those with motor

deficits.

2 Human State Estimation

Biomedical signal analysis is nowadays a method of the greatest importance for data

interpretation in medicine and biology, providing vital information about the condition

and affective/emotional states of subjects. The demand for a correct and prompt diag-

nosis leads to a mandatory identification of insufficiency signs in the clinical context

(Kayyali et al., 2008). Consequently, to analyze and follow up a subject’s condition it

is very important to monitor and visualize the acquired signals and extract relevant in-

formation from them. In clinical cases, such as sleep disorders and neuromuscular dis-

eases, a constant monitoring of the patient’s condition is necessary (Pinto et al., 2010).

In patients with neuromuscular diseases, heart rate variability, respiration, muscular and

electrodermal activity signals are extremely important, since they indicate when a mus-

cular crisis is occurring.

Cameras have been used to detect and estimate the pose of human subjects (Lim et

al., 2013) and body parts (Girshick et al., 2011)], detect faces (Xiao et al., 2004) and

their expressions (Yang et al., 2008) and, at a close range, detect eye movement and

EPS Portugal 2017/2018 2017 - OPPORTUNITIES AND CHALLENGES for European Projects

gaze direction (Morimoto et al., 2005). Recently, with the massification of RGBD sen-

sors in the gaming business, new levels of precision and reliability are being achieved

in such measurements (Guha and Ward, 2012). The main advantage of model-based

approaches is that it can reliably handle occlusions, noise, scale and rotation very well

in contrast to the model-free approach (Zhang et al., 2007). The main advantages of

Model-free methods are their simplicity and speed. Hidden Markov Models have been

successfully used in gesture recognition (Saponaro et al., 2013). Goffredo et al. (2008)

introduced view-independent markerless gait analysis based on the anthropometric

propositions of human limbs and the characteristics of gait.

3 Motor Training and Rehabilitation

The use of gaming approaches to motivate players to engage in physical activity is

popularly known as exergaming. Research on commercially available products has

shown that they can produce moderate to vigorous physical activity (Garn et al., 2012),

and that it results in physical, social, and cognitive benefits (Staiano et al., 2011). Un-

fortunately, these commercial tools are developed to target healthy young adults and

they are not suited to elderly or motor (re)training. Most rehabilitation treatments in-

volve repetitive exercises that are initially taught in a clinic and then continued at home.

Compliance with the regimen is critical for successful rehabilitation – both in terms of

adherence to the schedule of exercise and performing them correctly. A large percent-

age of people comply only partially, if at all, leading to minimal improvement or, at

worst, further injury. There exist a class of computer-based systems called Virtual

Coaches (VC) (Ding et al., 2010) that are aimed at mitigating the above mentioned

limitations using state-of-the-art technology to capture considerably more detailed data

regarding patient performance than previous experimental interventions. The benefits

brought by VC are multifold: (1) VC can incorporate scheduled exercises following

prescriptions of healthcare providers; (2) VC can monitor the performance of the user

and provide appropriate feedback and encouragement for training compliance; (3) pa-

tients can exercise independently at their homes after hospital discharge; and (4) care

providers can monitor remotely the progress of the user and upload new training pro-

tocols. Those approaches have been used in the past with success for rehabilitation,

exercise, proper use of assistive technology and accomplishing instrumental activities

of daily living (Siewiorek et al., 2012; Smailagic et al., 2013).

One of the latest approaches in the field of rehabilitation is the use of Virtual Reality

(VR). A number of studies have shown that this technology has a positive impact on

functional motor recovery (Laver et al., 2012). VR based rehabilitation systems can

support the requirements for effective motor (re)training (Cameirão et al., 2008). VR

based approaches allow for a combination of features including: low cost; personaliza-

tion of training; unsupervised training; goal-oriented actions; adaptability to a broad

range of patients; quantifiable outcome measures; extended feedback; and motivation

thanks to the use of game elements (Lucca, 2009).

Augmented Human Assistance (AHA) - Active aging Â˚u Serious games Â˚u Assistive robotics Â˚u Augmented reality Â˚u Virtual coach

4 Plan and Methods

The consortium of AHA includes research institutions and industry with expertise in

complementary areas of human motricity, robotics, human computer interaction, data

processing, bio-sensing technologies, and virtual and augmented reality solutions.

Moreover, the cross-disciplinary nature of the project requires the consortium members

to be organized in joint international teams across institutions.

Collaborations are organized in specific work packages that address complex real-

world research and technological challenges and have to deliver concrete building

blocks for the Robotic Assistance Platform (see Figure 1). Each work package has a

leading partner that will coordinate the activities and tasks of that work package. In this

project there will be an active involvement of the industrial partners not only as advi-

sors, providing know-how in business and innovation, but also as active contributors in

the realization of the technological and scientific work.

Fig. 1. Organization of the contributions of the different project work packages in the research

component of the AHA: Augmented Human Assistance project.

EPS Portugal 2017/2018 2017 - OPPORTUNITIES AND CHALLENGES for European Projects

5 Results

The principal achievements of the AHA project will be presented in the following five

sections: (1) Human State Awareness; (2) Augmented Reality Training; (3) Virtual

Coach; (4) Robotic Assistance Platform; (5) End user evaluation.

Human State Awareness

Physiological Sensing. The sensing system also includes a physiological platform, Bi-

osignalspluxTM available commercially. This system, as shown in Figure 2 (a), is a

wearable and wireless signal recorder that acquires several body signals. The radio

transmission is performed via Bluetooth to a computer or mobile using the Opensignals

open-source software. This software is versatile and scalable software for biosignals

visualization and analysis.

Fig. 2. (a) BiosignalspluxTM. (b) Signals acquired by the Biosignalsplux during an exercise.

ECG =Electrocardiogram, RESP = Respiratory rate, BVP = Blood Volume Pressure; ACC =

Accelerogram.

The signals depicted in Figure 2 (b) were acquired during an exercise trial and they

consist in four biosignals used in the context of AHA project: electrocardiogram (ECG),

which measures the different stages of the heart beat; respiration (RESP), measurement

of the periodic changes of the thorax movement while breathing; blood volume pressure

(BVP), an optical sensor that acquires the changes in volume of the capillaries, trans-

ducing the blood pulse; accelerometer (ACC) in three axis, recording the variation of

movements of the person in the different directions. Biosignalsplux has been used suc-

cessfully in several research and clinical applications (Barandas et al., 2015; Rebelo et

al., 2013).

Apart from these biosignals, bibliography has validated the grip strength as a meas-

urement of physical condition. This indicator may be used as a predictor for disability

and is considered to be useful for screening middle-aged and older adults (Bohannon,

Augmented Human Assistance (AHA) - Active aging Â˚u Serious games Â˚u Assistive robotics Â˚u Augmented reality Â˚u Virtual coach

2008). Therefore, in the context of this project and to assess physiological measure-

ments that could estimate the physical condition and evolution of the elderly user, a

new portable device was developed that measures the force applied by the grip. Since

the device is to be grasped, other three opportunistic signals can be extracted from the

hand without the use of gel electrodes. The extracted biosignals are not only BVP and

ACC, but also the electrodermal activity (EDA), which measures the changes in skin

conductivity controlled by the sympathetic nervous system. This gadget is ergonomic

and communicates wirelessly with Biosignalsplux, allowing for direct connection to

Opensignals for data visualization and analysis.

Signal analysis follows the acquisition, and relies on the morphological representa-

tion of the signal. Since the realization of the physical exercises require movement, the

signals may present noise and artifacts. The need to find clean areas for signal pro-

cessing resulted in two algorithms, one that identifies the noisy areas (Rodrigues et al.,

2017) and other that learns the signal morphology and replicates it for heavily corrupted

data (Belo et al., 2017).

Computer-based System for Assisting and Automating Functional Fitness Assessment.

The assessment of functional fitness components in older adults is important to identify

functional mobility disabilities and then targeted individualized exercise program. This

assessment is typically done through validated battery tests such as the Senior Fitness

Test (SFT; Rikli & Jones, 2013). The SFT is designed to be easy to administer while

not requiring extensive time, equipment or space. It is a valuable instrument for profes-

sional in evaluating and identifying risk factors, planning and assessing training pro-

grams, educating and setting goals. For the AHA human state awareness module we

developed a computer-based system for assisting and automating SFT administration

and scoring in the elderly population (Gonçalves et al., 2015). We considered the fol-

lowing domains and subtests of the SFT:

 Lower Body Strength: measured through the 30-second Chair-stand Test that

consists on counting the number of times a participant can fully stand and sit

from a chair, with the arms crossed, during a 30 seconds interval.

 Aerobic Endurance: assessed with the 2-minute Step Test. The test consists

on having the participant step in place for 2 minutes, raising the knees up to a

height marker placed halfway between the knee level and hip level. The num-

ber of times each knee reaches the target height is the score of the test.

 Agility and Dynamic Balance: measured with the 8-foot Up-and-go Test. In

this test, starting from a seated position, the user stands on a “go” signal, walks

2.4 m, turns around, walks back to the chair and sits. The participant practices

once and then perform two trials. The score is the fastest time of the two trials.

Our system automates scoring of these tests through the use of a Kinect V2 RGB-D

sensor for body tracking and gesture detectors for the evaluation of movement execu-

tion. Marker-less infrared systems present the lowest cost option for body tracking.

These devices estimate human body poses by analyzing the 3D depth information from

a scene while requiring minimal setup and no markers. They have been widely used in

research, for example, for designing full-body interactions in exergaming for older-

EPS Portugal 2017/2018 2017 - OPPORTUNITIES AND CHALLENGES for European Projects

adults (Gerling et al., 2012); for motion tracking in gait evaluation (Gabel et al., 2012;

Stone & Skubic 2011; Chaaraouiet al., 2015); as a guidance, correction and scoring

prototype for shoulder abduction exercises (Gama et al., 2012); for gesture detection

associated with muscle and joint pain, common in older-adults (Saha et al., 2013); or

as a tool to assist in the medical diagnosis and monitoring of Parkinson’s disease

through movement analysis (Spasojević et al., 2015).

The system was developed and trained with optimal data collected in laboratory con-

ditions [as shown in Figure 3 (a)], its performance was evaluated in a real environment

with 22 elderly end-users and compared to traditional SFT administered by an expert,

seen in Figure 3. Results show a high accuracy of our system in identifying movement

patterns (>95%) and consistency with the traditional fitness assessment method scores,

further details about the performance and results can be found in the authors’ original

publication (Gonçalves et al., 2015). In the Figure 3 b) it is possible to see a depth and

skeleton view of the Kinect V2 during an 8-ft Up-and-Go. The results suggest that the

technology is a viable option to support health and fitness professionals in the assess-

ment of physical function in the older population and could be deployed for at home

use in the context of fitness programs with the potential to be used autonomously by

non-experts.

a) b)

Fig. 3. Laboratory collected data being used to train a step detector for the 2-minute Step Test.

Augmented Reality Training

Exergames. Research has shown effectiveness and benefits of active-play with Exer-

games, which are videogames that use body movement as input control and promote

some form of exertion while playing, having impact in both physical and cognitive

functions (Molina, et al., 2014). A set of 4 Exergames was brainstormed, designed and

developed along various stages using human-centered techniques which involved the

Augmented Human Assistance (AHA) - Active aging Â˚u Serious games Â˚u Assistive robotics Â˚u Augmented reality Â˚u Virtual coach

end-users and health professionals (Gonçalves et al., 2017; Muñoz et al., 2018). The

duration of the process was 19 weeks divided in four main stages: conceptualization,

initial development, rapid contextual design and iteration, and polishing. ACSM’ rec-

ommendations were followed in the different fitness domains by means of offering

training personalization with fine game parametrization. Exercises incorporated in the

Exergames were developed by sport science professionals, which targeted dimensions

such as: motor ability (balance, agility and flexibility), aerobic endurance and muscular

strength (lower and upper limbs, and trunk).

Considering that the target population would feel more identified and engaged with

the Exergame experiences, the set of Exergames was inspired on a virtual tour in Por-

tugal. Traditional experiences were incorporated in the scenarios, such as miming the

smash of grapes for wine production, transporting the wine barrels along the river, ex-

periencing a toboggan ride in Madeira island, and playing piano on a fado house (Figure

4).

The developed games went through a process of multiple iterations involving the

target population, as well sports and health professionals and scientists, game designers

and developers, till being ready for a controlled longitudinal study aiming to assess

their effectiveness among the senior population. The whole design process is described

in (Muñoz et al., 2018).

Fig. 4. Screenshots of the 4 developed Exergames (Gonçalves et al., 2017).

Integrative System for Exergames. Due to the highly configurable nature of Exergames,

managing and preparing training sessions can be difficult and time consuming when

faced with complex and a large amount of parameter choices. An integrative system

that can aggregate independent Exergames, such as the ones previously presented, was

also ideated and designed using human-centred techniques, involving the main pro-

spective end-users of the configurable User Interface (UI), namely health and sports

professionals (Paulino et al., 2018). This integrative system will allow configuring and

managing the different Exergames through a common UI. Data derived from physical

and cognitive assessments from users will serve as input to provide decision support on

the creation of training/rehabilitation plans adapted for each end-user profile. Data from

the training sessions will be hosted both locally and in a cloud database enabling easy

access and meaningful visualization of the historical progress of end-users (Figure 5).

The initial steps carried out to design the integrative system were divided in 3 major

stages: requirements engineering, software design, and human-computer interaction.

Techniques such as semi-structured interviews, card sorting, and paper prototyping

were used to involve the main interactors of the envisioned system in the design process

allowing to design with the consideration of their preferences and needs. Future work

will involve a usability study after the basic UI implementation.

EPS Portugal 2017/2018 2017 - OPPORTUNITIES AND CHALLENGES for European Projects

Fig. 5. Integrative system's overview.

PEPE (The Portable Exergame Platform for Elderly). In this section, we describe the

design and test of PEPE - The Portable Exergame Platform for Elderly. This is a hard-

ware and software system that incorporates the data acquisition, augmented reality se-

rious games (floor projections) in a wheeled base and foldable mechanical structure.

This platform does not contain the autonomous robotics components that, due to cost,

is inaccessible to many institutions. Instead, the platform can be manually moved by

the professionals in the institutions and serve users either in public or private rooms.

Our design approach was based on user-centered design (Abras et al., 2004), accord-

ing to the following three stages.

 On the first stage, we defined the main requirements: software and hardware

specifications to execute the Exergames and the easy operation and deploy-

ment within a typical elderly care institution.

 In the second stage, we visited the institutions with an early prototype. This

prototype was used to do thematic analysis (Braun & Clarke, 2006), using

quantitative and qualitative data extracted through questionnaires and semi-

structured interviews regarding functional aspects, possible usability, appear-

ance, and physiological measurement requirements. The intention of this pro-

cess was to involve the people for whom we are developing in an iterative

process (Baek et al., 2008). To collect a large heterogeneous users sample, our

study included an array of multidisciplinary professionals (p=6) and people

from the geriatric segment (u=24) across three different institutions. Regard-

ing the appearance of PEPE, there was no consensus among the interviewee's

sample. We proposed several concepts using bidimensional sketches and tri-

dimensional modeling [Figure 6, (a); (b) and (c)] to explore different alterna-

tives.

 On the third stage, we repeated the second phase focusing on the alternative

concepts that we did for the cover and interfaces, based on the previous results

coming from the thematic analysis. We used PEPE as a way to study its pur-

pose as a motivational and physical exercise elicitation device and also as a

validation platform to test some components that can be implemented later on

the robot Vizzy.

Augmented Human Assistance (AHA) - Active aging Â˚u Serious games Â˚u Assistive robotics Â˚u Augmented reality Â˚u Virtual coach

a) b) c)

Fig. 6. Appearance of PEPE: bidimensional sketches and tridimensional modeling

Following the results of our studies, we have to converged to the design of PEPE shown

in (Figure 7). The core aspects leading to the final design were related to the mandatory

empathic aesthetics of the platform, simultaneous usability for the seniors and

healthcare professionals and, the adaptation/customization of the Exergames to the

multiple constraints that seniors might have that increase success rates (Harriette,

2013). The platform allows Exergames to be played not only by healthy people but also

by people with some physical and/or cognitive impairment. We also can extend the

number of people that interact with the platform, since PEPE is capable of going near

people who can't move.

Fig. 7. PEPE Platform.

PEPE was also seen as a dynamizing tool that can optimize the traditional process inside

institutions, being helpful to Physiotherapists, Psychologists, and Occupational Thera-

pists during their performances. Our findings show that users can be proactive in en-

gaging in physical exercise with PEPE thus serving the purpose of sedentary preven-

tions and rehabilitation while being also entertaining. Further information regarding the

results of this section can be found in the author's original publication (Simao & Ber-

nardino, 2017).

Virtual Coach

A Virtual Coach is an attentive personalized system that monitors the context of a user

to provide a feedback or assistance. It leverages machine learning with sensor data to

understand user contexts (e.g. locations, physical activities, emotions, and social). For

EPS Portugal 2017/2018 2017 - OPPORTUNITIES AND CHALLENGES for European Projects

instance, a Virtual Coach can monitor repetitive rehabilitation exercises and assess pa-

tient’s exercises performance and provide corrective feedback. This feedback can en-

sure the correct execution of rehabilitation exercises and motivate patient’s engage-

ment. In addition, therapists can judge the quality of in-home rehabilitation with quan-

titative performance data. We first describe our methods to develop a Virtual Coach for

stroke rehabilitation exercises and then demonstrate its capabilities.

Methodology. A Virtual Coach system assists a patient to engage correctly in rehabili-

tation exercises with minimal supervision of a therapist. To mimic a therapist’s vision-

based assessment of rehabilitation exercises, the Virtual Coach utilizes a Kinect sensor

to track kinematic positions of a patient and employs machine-learning algorithms to

provide assessment and guidance

Given an exercise trial, the system assesses a performance score along with corrective

feedback on joint positions. The performance score will be stored to record a patient’s

progress. The assessment module quantifies the quality of an exercise and collects per-

formance data. First, we analyzed the existing manual clinical stroke assessment tools

(i.e. Fugl Meyer Assessment and Wolf Motor Function Tests) and in discussions with

therapists developed three performance metrics of stroke rehabilitation: ‘Precision’,

‘Smoothness’, and ‘Compensation’. The ‘Precision’ metric represents how accurately

an exercise is performed. The ‘Smoothness’ metric indicates the level of trembling. The

‘Compensation’ metric checks whether a patient involves any compensatory move-

ments (e.g. leaning torso forward).

The Virtual Coach system extracts normalized kinematic features and applies Deep-

Reinforcement Learning to identify important features. The system then trains binary

(correct, incorrect) classifiers to classify the correctness of an exercise movement in

terms of the three performance metrics. Utilizing these classifiers, the system can quan-

tify the probability of being a correct movement yielding a performance score.

The guidance module learns sequential joint positions of an exercise and generates cor-

rective high-level instructions (e.g. move your elbow upward) upon detecting an incor-

rect movement. The system utilizes grid-representation to describe an exercise as se-

quential pairs of grid-based current and next joint locations. It then trains a model of

sequential joint movements to instruct a patient how to correctly place joint positions.

Experiments and Results. To validate the feasibility of a Virtual Coach system, we uti-

lize three upper-limb stroke rehabilitation exercises: Exercise 1 (E1) – ‘Bring a cup to

the mouth’, Exercise 2 (E2) – ‘Switch a light on’, and Exercise 3 (E3) – ‘Move forward

a cane’. These exercises are selected due to their correspondence with major motion

patterns: elbow flexion for E1, shoulder flexion for E2, and elbow extension for E3.

For the data collection, we recruited 15 post-stroke survivors with different level of

functional abilities and 11 healthy subjects. A post-stroke survivor performed 10 repe-

titions of each exercise on both affected and unaffected sides. A healthy subject per-

formed 15 repetitions of each exercise using their dominant side. In total, each upper

limb exercise contains 465 trials: 315 trials of correct movements and 150 trials of af-

fected movements. A therapist observed recorded videos of exercise trials and evalu-

ated a score of the three performance metrics on 4-point ordinal scale (0-3).

For classification models, we used Decision Tree (DT) and Hidden Markov Model

(HMM) as shown in Table 1. After comparing classification accuracy, we selected DTs

Augmented Human Assistance (AHA) - Active aging Â˚u Serious games Â˚u Assistive robotics Â˚u Augmented reality Â˚u Virtual coach

for the ‘Precision’ and ‘Compensation’ metrics and HMMs for the ‘Smoothness’ met-

ric. Our assessment module achieves 83.65 – 93.46% agreement with therapist’s obser-

vation scores for an individual performance metric over the three exercises. The assess-

ment module has an average 78.45% agreement and 0.8223 Pearson’s Correlation Co-

efficient (p < 0.001) for all three metrics.

Table 1. Classification Accuracy of Three Performance Metrics with DTs and HMMs.

Conclusion and Discussion. Given a set of rehabilitation exercises prescribed by a ther-

apist, a Virtual Coach can use machine learning techniques to evaluate the quality of a

movement and generate performance and corrective feedback advice. Our experiments

demonstrate the feasibility to learn therapist’s assessment and automatically quantify

the performance of rehabilitation exercises. Thus, a Virtual Coach has potential to en-

hance patient’s independent engagement in rehabilitation exercises after hospital dis-

charge. From a therapist’s perspective, this technology can support remote monitoring

of patient’s progress. For integration with other work packages in the AHA project, we

developed communication channels with a mobile robotic platform and Augmented

Reality (AR) training module. The Virtual Coach can support patients or elderly per-

sons from providing simple reminders to guidance throughout an exercise. Leveraging

the user state analysis from the Monitoring Module (WP1) and automated assessment,

the Virtual Coach can recommend performing more personalized serious games. The

current implementation primarily focuses on guidance during an exercise. In the future,

it would be interesting to explore the application of a Virtual Coach on more diverse

and complex tasks than repetitive exercises.

Robotic Assistance Platform

On this project, we use the Vizzy robot (Moreno et al., 2016) as a Socially Assistive

Robot (SAR) that plays a coaching role during physical exercise and also serves as an

assistant for professionals. For a robot coach to be successful, the interaction must be

pleasant, and people should perceive it as competent and trustworthy. To evaluate Viz-

zy's fitness for this role, study current limitations, and experience unforeseen interac-

tions, we deployed it in three elderly care centers in Portugal (LATI - Liga dos Amigos

Metrics

Algorithm

Exercise 1

Exercise 2

Exercise 3

Precision

94.12 ± 1.84 %

98.57 ± 0.50 %

92.26 ± 2.02 %

HMM

81.90 ± 2.97 %

90.50 ± 2.24 %

72.46 ± 3.41 %

Smoothness

56.85 ± 3.84 %

65.67 ± 3.63 %

63.68 ± 3.71 %

HMM

82.55 ± 2.90 %

82.09 ± 2.92 %

82.16 ± 2.94 %

Compensation

98.47 ± 0.94 %

97.41 ± 1.19 %

93.55 ± 1.87 %

HMM

78.62 ± 2.8 %

80.99 ± 3.00 %

76.34 ± 3.22 %

EPS Portugal 2017/2018 2017 - OPPORTUNITIES AND CHALLENGES for European Projects

da Terceira Idade, Centro Social Comunitário da Nossa Sra. dos Milagres, and Residên-

cia Sénior de Belverde). The robot coached a total of 36 seniors (aged between 65 and

94 years old, μ = 80.83, 𝛔²=5.84), inviting, engaging, instructing, and providing moti-

vational feedback while they played the ExerPong exergame. Since dialogue and head

gestures are not yet fully automated, we controlled the robot via a Wizard-of-Oz inter-

face. Next, we will describe (i) Vizzy and the technological implementations for the

experiment, (ii) the user study, (iii) results and lessons learned.

Technical Details. Vizzy is a general purpose SAR platform with an anthropomorphic

upper torso, and biologically inspired head and eye movements (Roncone et al., 2016).

Its two arms allow it to perform non-verbal communicative gestures familiar to humans

(although still not used in this study). Vizzy uses a mobile Segway platform to navigate

autonomously in known areas. Its front and rear lasers allow it to avoid obstacles along

its way while localizing itself on a known map. Vizzy has two RGB cameras on its eyes

and an RGB-D sensor on its torso that can be used to detect people, objects and obsta-

cles not captured by lasers. The robot can also emit sounds and synthesize speech.

Given the unpredictability of the experimental setup, WoZ interfaces needed to be

fast, robust and easy to use. For this purpose, we used two interfaces (Figure 8): a cus-

tom-made dialog control interface that can be easily accessed using any web supporting

device (but optimally used on a tablet) and a motor control interface using Rviz with

custom plugins. To control the robot's gaze and movements we developed two plugins

for Rviz: ClickableGazeDisplay and WASDTeleop. The ClickableGazeDisplay lets the

"wizard" select the gaze point by clicking on the camera image. The WASDTeleop

allows the direct control of wheel velocities using the W, A, S, and D keyboard keys.

These plugins' code is open source can be easily modified for other robots, as needed.

Fig. 8. Wizard of Oz interfaces.

Experiment. During the experiment (Figure 9), the robot approached a senior user and

invited her/him to play an Exergame. If the person accepted, the robot would guide

him/her towards the PEPE platform. Then, the robot introduced the game giving in-

structions on how to play. During the game, the robot also assists the person with cor-

rective instructions if necessary. A second role of the robot is engaging the person dur-

ing the game by providing feedback, and by assessing interactively with the person if

Augmented Human Assistance (AHA) - Active aging Â˚u Serious games Â˚u Assistive robotics Â˚u Augmented reality Â˚u Virtual coach

the game can continue or if it should stop. Afterwards, each participant answered a

questionnaire after taking a picture with the robot.

The questionnaire was composed of five-point Likert scale items ("totally disagree"

= 1, "totally agree" = 5), adapted from the Godspeed Questionnaire (Bartneck et al.,

2009), the ALMERE model (Heerink et al., 2010) and scales proposed by Jian et al.,

2000.

Fig. 9. Experiment steps.

Results and Lessons Learned. Users reported high perceived enjoyment of playing Ex-

erPong with Vizzy (μ = 4.67, 𝛔²=0.47), high perceived robot competence (μ = 4.47,

𝛔²=0.41) and high perceived trust (μ = 4.36, 𝛔²=0.79). On average, people liked the

robot (μ = 4.56, 𝛔²=0.71) and thought it was cute (μ = 4.36, 𝛔²=0.81), but thought it

looked artificial (μ = 3.78, 𝛔²=1.66) and had a machine-like appearance (μ = 2.39,

𝛔²=2.016). They also found the robot’s movements elegant (μ = 3.81, 𝛔²=0.78). All

these items were statistically different than the neutral value when performing a One-

Sample T-Test (p<0.05 for the human-like - machine-like appearance item, and

p<0.005 for the remaining items).

Informal observations and interviews also revealed interesting insights that were us-

ing for fine-tuning the system and that suggest future improvements. For instance, ac-

curate gaze direction is fundamental for interaction, because otherwise people will be

EPS Portugal 2017/2018 2017 - OPPORTUNITIES AND CHALLENGES for European Projects

confused, and the robot must use short and assertive utterances during gameplay. More

information can be found on the authors’ workshop paper (Avelino et al., 2018).

End User Evaluation

Adding Physiological Intelligence to Exergames. By performing a pilot study to inves-

tigate the cardiovascular mechanisms that acted during the interaction with a customi-

zable Exergame, we concluded that important heart rate (HR) and heart rate variability

(HRV) responses could be modulated by means of adjusting game difficulty parameters

(Muñoz et al., 2016). The next step in this research was the idea to create an intelligent

adaptation for the Exerpong based on cardiovascular measurements in real time by

means of combining two approaches: i) a construct from the physiological computing

field called the biocybernetic loop (Pope et al., 2014) and ii) the well-established aero-

bic training method based on targeted HR levels (Heyward & Gibson, 2014).

Target HR: by following the ACSM guidelines for older adults (Jones et al., 2005),

aerobic exercises should be controlled in regards of the intensities levels guaranteeing

that older users can maximize the benefits of stressing the heart without over exercising

it. This zone of healthy exercise intensity can be defined by means of HR measurements

that uses a percentage of the HR reserve (HRR) – which is the difference between max-

imum HR (HR

max

) and HR during resting (HR

rest

) – as expressed in the equation:

Target HR = [% exercise intensity * (HR

max

– HR

rest

)] + HR

rest

This target zone is individual for each user and it is recommended older adults to exert

under 40 % to 70 % of their target HR (Jones et al., 2005).

Biocybernetic Loop: this concept relies on the detection of human states based on phys-

iological sensing in order to modify the system behavior in real time. By using novel

and minimally intrusive wearable sensors, HR and HRV measurements can be recorded

with high levels of accuracy during exercise routines. HR data in real time can be used

to modify game parameters in order to keep users exercising at desirable intensities, the

targeted HR levels in this case.

To evaluate the effectiveness of a Cardio-Adaptive Exergame based on biocyber-

netic adaptation that uses the target HR approach, we carried out a within subjects ex-

periment comparing our adaptive training routine against conventional group fitness

sessions leader by sports science professionals.

Cardio-adaptive Exerpong Approach

Exergame Design and Setup: a customizable Exergame inspired in the classic 2D pong

was created: the Exerpong. Players are challenged to hit a ball using a virtual paddle,

which is mapped to the player’s waist position via the Kinect V2 sensor (Microsoft,

Washington, USA). Exerpong was designed and developed in the game engine

Unity3D (Unity Technologies, San Francisco, USA) which conveniently allows a com-

plete game customization and data logging. Exerpong allows a physical training of bal-

ance and agility while the physiological adaptation is oriented to maximize aerobic per-

formance. The Exerpong is projected on the floor on a white 2.5 x 3.0 surface (see

Figure 10).

Augmented Human Assistance (AHA) - Active aging Â˚u Serious games Â˚u Assistive robotics Â˚u Augmented reality Â˚u Virtual coach

Fig. 10. Image showing the final setup of the Exerpong.

HR Data: to record the cardiovascular responses, we used a Motorola 360 smartwatch

which streams computed HR data with a 1 Hz sampling frequency to a custom-made

software called the Biocybernetic Loop Engine (Muñoz et al., 2017), which is in charge

of creating the physiological adaptation and the communication with the Exerpong.

Adaptation Rules: to help players reach their individual target HR levels while keeping

them motivated to exercise, the Cardio-Adaptive Exerpong uses a dual adaptation rule

(Sinclair et al., 2009).

Gameplay Adaptation: the paddle size increase once players miss balls and vice versa,

the ball velocity automatically decreases if the player misses three consecutive balls.

HR-based Adaptation: the ball velocity increases if the 30 seconds HR average is under

the target HR and decreases it otherwise.

Preliminary Results. By analyzing the time fifteen older adults (11 females, ages 66 ±

7 years) spent in their individual targeted zones during 20 minutes of exercise with the

Cardio-Adaptive Exerpong, we demonstrated that our biocybernetic system led players

to exert around 40% more time in this recommended zone once compared with con-

ventional training. Moreover, we also showed the feasibility of use the HR data from

the smartwatch to assess HRV changes during the interaction with the adaptive system

(Muñoz et al., 2017).

Finally to extend our approach, we are currently carrying out a 6-weeks study aiming

at quantify the long-term effects of training with the Cardio-Adaptive Exerpong in a

local senior gymnasium.

6 Conclusions

A new generation of ICT based solutions for promotion of physical exercise in the older

population, either for the prevention of inactivity-related diseases or for the rehabilita-

tion of motor deficits are developed with AHA project. This is very important to better

understand the end-user challenges when engaging with technological solutions for

EPS Portugal 2017/2018 2017 - OPPORTUNITIES AND CHALLENGES for European Projects

physical activities, namely, the combination of customized augmented reality games

and assistive robot coaching. The three main contribution of AHA project are: (a) a

Mobile Augmented Reality Platform that projects Serious Games in the environment

for the training of several fitness dimensions in the older persons (balance, mobility,

agility, strength, endurance, etc); (b) a set of human-robot interaction modalities to en-

gage and motivate users in the exercises, and (c) a set of automated senior fitness tests

to assess the functional fitness levels of the users combined with a web-based platform

to create and store the users’ profiles.

Acknowledgements. This work is supported by the Portuguese Foundation for Science

and Technology through the Augmented Human Assistance project

(CMUPERI/HCI/0046/2013). Authors would like to thank the valuable collaboration

of the following collaborators: Honorato Sousa; Lúcia Conceição; Plinio Moreno; Rui

Figueiredo; Nuno Duarte; Ricardo Nunes; Ricardo Ribeiro and Jose Santos-Victor. Au-

thors would also to thank the following partnerships: Câmara Municipal do Funchal

(Ginásios de Santo António e São Martinho); LATI - Liga dos Amigos da Terceira

Idade; Residencia Senior Belverde; Caritas Diocesiana Coimbra – Centro; Social Co-

munitário da Nossa Sra. Dos Milagres; Andreia Cordeiro; Ana Luisa Jegundo, Ana

Mendes, Carla Cortes, Flávia Rodrigues, Gonçalo Martins, Luis Santos. Finally, au-

thors are especially grateful to the older people (end-users) for their participation and

interest.

References

1. World Health Organization. (2012). World health statistics 2012. Genebra, Switzerland:

World Health Organization Press.

2. Bemelmans, R., Gelderblom, G.J., Jonker, P., & de Witte, L. (2012). Socially assistive ro-

bots in elderly care: A systematic review into effects and effectiveness. Journal of the Amer-

ican Medical Directors Association, 13(2):114-120

3. Matarić, M.J., Eriksson, J., Feil-Seifer, D., & Winstein, C.J. (2009). Socially assistive ro-

botics for post-stroke rehabilitation". Journal of neuroengineering and rehabilitation, 4:5.

4. Broekens, J., Heerink, M., & Rosendal, H. (2009). Assistive social robots in elderly care: a

review. Gerontechnology, 8(2):94-103.

5. Scassellati, B. (2007). How social robots will help us to diagnose, treat, and understand au-

tism. Robotics research, 552-563.

6. Kidd, C.D., & Cynthia, B. (2008) Robots at home: Understanding long-term human- robot

interaction. Intelligent Robots and Systems, 2008. IROS 2008. IEEE/RSJ International Con-

ference on IEEE.

7. Pineau, Joelle, P., Montemerlo, M., Pollack, M., Roy, N., & Thrun, S. (2003). Towards ro-

botic assistants in nursing homes: Challenges and results. Robotics and Autonomous Sys-

tems 42(3):271-281

8. Rincon, F., Vibbert, M., Childs, V., Fry, R., Caliguri, D., Urtecho, J., et al., (2012). Imple-

mentation of a model of robotic tele-presence (RTP) in the Neuro-ICU: effect on critical

care nursing team satisfaction. Neurocritical care, (1):97-101.

Augmented Human Assistance (AHA) - Active aging Â˚u Serious games Â˚u Assistive robotics Â˚u Augmented reality Â˚u Virtual coach

9. Kayyali, H., Weimer, S., Frederick, C., Martin, C., Basa, D., Juguilon, J., & Jugilioni, F.

(2008). Remotely attended home monitoring of sleep disorders. Telemedicine journal and

e-health, 14(4):371–4.

10. Pinto, A., Almeida, J. P., Pinto, S., Pereira, J., Oliveira, A.G., & de Carvalho, M. (2010).

Home telemonitoring of non-invasive ventilation decreases healthcare utilisation in a pro-

spective controlled trial of patients with amyotrophic lateral sclerosis. Journal of Neurology,

Neurosurgery & Psychiatry, 81(11):1238-42.

11. Lim, J.J., Zitnick, C.L., & Dollár P. (2013). Sketch Tokens: A Learned Mid- level Repre-

sentation for Contour and Object Detection." CVPR, 2013. IEEE Conference on Computer

Vision and Pattern Recognition,

12. Girshick, R.B., Felzenszwalb, P.F., & Mcallester, D.A. (2011). Object detection with gram-

mar models. Advances in Neural Information Processing Systems. Available at:

http://cs.brown.edu/people/pfelzens/papers/grammar-nips11.pdf.

13. Yang, Y., Ge, S.S., & Wang, L.C. (2008). Facial expression recognition and tracking for

intelligent human-robot interaction. Intelligent Service Robotics,1,2: 143-157.

14. Xiao, R., Li, M., & Zhang, H. (2004). Robust multipose face detection in images. Circuits

and Systems for Video Technology, IEEE Transactions on, 14,1: 31-41.

15. Morimoto, C.H., & Mimica M. (2005). Eye gaze tracking techniques for interactive appli-

cations. Computer Vision and Image Understanding, 98,1: 4-24.

16. Guha, T., & Ward, R.K. (2012). Learning Sparse Representations for Human Action Recog-

nition. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 34, 8:1576,1588.

17. Zhang, R., Vogler, C.H., & Metaxas, D. (2007). Human gait recognition at sagittal plane.

Image and Vision Computing, 25, 3:321–30.

18. Saponaro, G., Salvi, G., & Bernardino, A. (2013). Robot Anticipation of Human Intentions

through Continuous Gesture Recognition, In International Conference on Collaboration

Technologies and Systems (CTS). doi: 10.1109/CTS.2013.6567232.

19. Goffredo, M., Seely, R.D., Carter, J.N., & Nixon, M.S. (2008). Markerless view independent

gait analysis with self-camera calibration, In Proceedings of International Conference on

Automatic Face and Gesture Recognition. doi: 10.1109/AFGR.2008.4813366

20. Garn, A.C., Baker, B.L., Beasley, E.K., & Solmon, M.A. (2012). What are the benefits of a

commercial exergaming platform for college students? Examining physical activity, enjoy-

ment, and future intentions. Journal of physical activity & health, 9, 2:311.

21. Staiano, A.E., & Sandra L.C. (2011). Exergames for physical education courses: Physical,

social, and cognitive benefits. Child Development Perspectives, 5,2: 93- 98.

22. Ding, D., Liu, H.Y., Cooper, R., Cooper, R.A., Smailagic, A., & Siewiorek, D. (2010). Vir-

tual Coach Technology for Supporting Self-Care. Physical Medicine and Rehabilitation

Clinics of North America, 21(1), 179-194.

23. Siewiorek, D., Smailagic, A., & Dey, A. (2012). Architecture and Applications of Virtual

Coaches. In Proceedings of the IEEE, Special Issue on Quality of Life Technology.

Doi:10.1109/JPROC.2012.2200560.

24. Smailagic, A., Siewiorek, D., Chakravarthula, S., Kar, A., Jagdale, N., Gautam, S. … Jagtap,

S. (2013). Emotion Recognition Modulating Behavior of Intelligent Systems, Proc. IEEE

International Symposium on Multimedia, IEEE Computer Society Press. Doi:

10.1109/ISM.2013.72

25. Laver, K., George, S., Thomas, S., Deutsch, J.E., & Crotty, M. (2012). “Cochrane review:

virtual reality for stroke rehabilitation, European journal of physical and rehabilitation,

48(3):523-30.

26. Cameirão, M.S., Bermúdez, S., & Verschure, P. (2008). Virtual reality based upper extrem-

ity rehabilitation following stroke: A review. Journal of CyberTherapy & Rehabilitation,

1(1).

EPS Portugal 2017/2018 2017 - OPPORTUNITIES AND CHALLENGES for European Projects

27. Lucca, L.F. (2009). Virtual reality and motor rehabilitation of the upper limb after stroke: a

generation of progress? Journal of rehabilitation medicine, 41(12):1003-100.

28. Barandas, M., Gamboa, H., & Fonseca, J.M. (2015) A real time biofeedback system using

visual user interface for physical rehabilitation, Procedia Manufacturing 3, 823-828.

29. Rebelo, D., Amma, C., Gamboa, H. & Schultz, T. Human Activity Recognition for an Intel-

ligent Knee Orthosis. In Proc. of the Int’l Conf. on Bio-inspired Systems and Signal Pro-

cessing (BIOSIGNALS), pp. 368-371. Available at: http://www.plux.info/files/ftp/docs/

RebeloAmmaGamboaSchultz_Biosignals2013.pdf.

30. Bohannon, R. W. (2008). Handgrip dynamometry predicts future outcomes in aging adults.

Journal of geriatric physical therapy, 31(1), 3-10.

31. Rodrigues, J., Belo, D., & Gamboa, H. (2017). Noise detection on ECG based on agglom-

erative clustering of morphological features. Computers in biology and medicine, 87, 322-

334.

32. Belo, D., Rodrigues, J., Vaz, J. R., Pezarat-Correia, P., & Gamboa, H. (2017). Biosignals

learning and synthesis using deep neural networks. Biomedical engineering online, 16(1),

115.

33. Rikli, R. E., & Jones, C.J. (2013). Senior Fitness Test Manual. Champaign, IL: Human Ki-

netics.

34. Gonçalves, É. Gouveia, M. Cameirão, & Badia, S.B. (2015). Automating Senior Fitness

Testing through Gesture Detection with Depth Sensors,” in Proceedings of the IET Interna-

tional Conference on Technologies for Active and Assisted Living (TechAAL 2015), Lon-

don, United Kingdom.

35. Gerling, K., Livingston, I. Nacke, L., & Mandryk, R. (2012). Full-body Motion-based Game

Interaction for Older Adults,” in Proceedings of the SIGCHI Conference on Human Factors

in Computing Systems, New York, NY, USA, pp. 1873–1882.

36. Gabel, M., Gilad-Bachrach, R. Renshaw, E. & Schuster, A. (2012). Full body gait analysis

with Kinect, in 2012 Annual International Conference of the IEEE Engineering in Medicine

and Biology Society (EMBC), pp. 1964–1967.

37. Stone E.E., & Skubic, M. (2011). Passive in-home measurement of stride-to-stride gait var-

iability comparing vision and Kinect sensing,” in 2011 Annual International Conference of

the IEEE Engineering in Medicine and Biology Society, EMBC, pp. 6491–6494.

38. Chaaraoui, J., Lopez, P., & Revuelta, F. (2015). Abnormal gait detection with RGB-D de-

vices using joint motion history features. presented at the 1st International Workshop on

Understanding Human Activities through 3D Sensors (UHA3DS’15), Ljubljana, Slovenia.

39. Gama, A., Chaves, T., Figueiredo, L., & Teichrieb, V. (2012). Guidance and Movement

Correction Based on Therapeutics Movements for Motor Rehabilitation Support Systems,”

in 2012 14th Symposium on Virtual and Augmented Reality (SVR), pp. 191–200.

40. Saha, S., Pal, M., Konar, A., & Janarthanan, R. (2013). Neural Network Based Gesture

Recognition for Elderly Health Care Using Kinect Sensor,” in Proceedings of the 4th Inter-

national Conference on Swarm, Evolutionary, and Memetic Computing - Volume 8298,

New York, NY, USA, pp. 376–386.

41. Spasojević, S., Santos-Victor, J. Ilić, T. Milanović, S. Potkonjak, V. & Rodić A. (2015). A

Vision-Based System for Movement Analysis in Medical Applications: The Example of

Parkinson Disease,” in Computer Vision Systems, Springer International Publishing, pp.

424–434.

42. Molina, K.I., Ricci, N.A. de Moraes, S. A. & Perracini, M.R. (2014). Virtual reality using

games for improving physical functioning in older adults: a systematic review. Journal of

neuroengineering and rehabilitation, 11:156. doi: 10.1186/1743-0003-11-156.

Augmented Human Assistance (AHA) - Active aging Â˚u Serious games Â˚u Assistive robotics Â˚u Augmented reality Â˚u Virtual coach

43. Gonçalves, A., Muñoz, J., Gouveia, E.R., Cameirão, M. & Bermúdez, S.B. (2017). Portu-

guese Tradition Inspired Exergames for Older People Strategic Tools to Promote Functional

Fitness. In Extended Abstracts - AHA icSports 2017, pp. 6-7.

44. Muñoz, J.E., Gonçalves, A., Gouveia, E.R., Cameirão, M.S., Bermúdez, S.B. (2018). Les-

sons learned from gamifying functional fitness training through hu-man-centered design

methods in Portuguese older adults. Games Health J. (in press).

45. Paulino, T., Muñoz, J. Bermudez, S., Cameirão, M.S. (2018). Design of an Integrative Sys-

tem for Configurable Exergames Targeting the Senior Population. In International Confer-

ence on Human Systems Engineering and Design: Future Trends and Applications (IHSED

2018), Reims Champagne-Ardenne, France.

46. Abras, C., Maloney-Krichmar, D., & Preece, J. (2004). User-Centered Design. Encyclopedia

of Human-Computer Interaction. Available at: http://www.elearning.co.il/home/pdf/4.pdf

47. Braun, V., & Clarke, V. (2006). Using thematic analysis in pscychology. Qualitative Re-

search in Pscychology. Qualitative Research in Psychology, 3(2), 77–101.

http://doi.org/10.1191/1478088706qp063oa

48. Baek, E. O., Cagiltay, K., Boling, E., & Frick, T. (2008). User-centered design and devel-

opment. Handbook of Research on Educational Communications and Technology, 14(1),

659–670. http://doi.org/10.1145/1273961.1273973

49. Harriette, H. (2013). Having It Their Way: The Big Opportunity In Personalized Products.

Retrieved February 25, 2017, from http://www.forbes.com/sites/baininsights/2013

/11/05/having-it-their-way-the-bigopportunity- in-personalized-products/

50. Simao, H. & Bernardino, A. (2017). User Centered Design of an Augmented Reality Gam-

ing Platform for Active Aging in Elderly Institutions. ICSports 151-162.

10.5220/0006606601510162.

51. Moreno, P., Nunes, R., Figueiredo, R., Ferreira, R., Bernardino, A., Santos-Victor, J. …

Aragão, M. (2016). Vizzy: A humanoid on wheels for assistive robotics. In Robot 2015:

Second Iberian Robotics Conference (pp. 17-28). Springer, Cham.

52. Roncone, A., Pattacini, U., Metta, G., & Natale, L. (2016). A Cartesian 6-DoF Gaze Con-

troller for Humanoid Robots. In Robotics: Science and Systems. Available at:

http://www.roboticsproceedings.org/rss12/p22.pdf.

53. Bartneck, C., Kulić, D., Croft, E., & Zoghbi, S. (2009). Measurement instruments for the

anthropomorphism, animacy, likeability, perceived intelligence, and perceived safety of ro-

bots. International journal of social robotics, 1(1), 71-81.

54. Heerink, M., Kröse, B., Evers, V., & Wielinga, B. (2010). Assessing acceptance of assistive

social agent technology by older adults: the almere model. International journal of social

robotics, 2(4), 361-375.

55. Jian, J. Y., Bisantz, A. M., & Drury, C. G. (2000). Foundations for an empirically determined

scale of trust in automated systems. International Journal of Cognitive Ergonomics, 4(1),

53-71.

56. Avelino, J., Simão, H., Ribeiro, R., Moreno, P., Figueiredo, R., Duarte, N., ... & Odekerken-

Schröder, G. (2018). Experiments with Vizzy as a Coach for Elderly Exercise. PREC

workshop at HRI 2018.

57. Muñoz, J. E., Cameirão, M. S., Rubio, E., Paulino, T., & i Badia, S. B. (2016). Modulation

of Physiological Responses and Activity Levels During Exergame Experiences. In 18th In-

ternational Conference on Virtual Worlds and Games for Serious Applications. IEEE.

58. Pope, A. T., Stephens, C. L., & Gilleade, K. (2014). Biocybernetic adaptation as biofeedback

training method. In Advances in physiological computing (pp. 91-115). Springer, London.

59. Heyward, V. H., & Gibson, A. (2014). Advanced fitness assessment and exercise prescrip-

tion 7th edition. Human kinetics.

EPS Portugal 2017/2018 2017 - OPPORTUNITIES AND CHALLENGES for European Projects

60. Jones, J., & Rose, R. 2005. Physical activity instruction of older adults. Champaign, IL:

Human Kinetics.

61. Muñoz, J.E., Rubio, E., Cameirao, M., & Bermúdez, S. (2017). The Biocybernetic Loop

Engine: an Integrated Tool for Creating Physiologically Adaptive Videogames. In 4th Inter-

national Conference in Physiological Computing Systems. Presented at the PhyCS.

62. Sinclair, J., Hingston, P., & Masek, M. (2009). Exergame development using the dual flow

model. In Proceedings of the Sixth Australasian Conference on Interactive Entertainment

(p. 11). ACM.

63. Muñoz, J. E., Gouveia, E. R., Cameirão, M. S., & i Badia, S. B. (2017). Heart Rate Varia-

bility in Exergaming. Feasibility and Benefits of Physiological Adaptation for Cardiorespir-

atory Training in Older Adults by Means of Smartwatches. In 5th International Congress on

Sport Science Research and Technology Support, INSTICC.

Augmented Human Assistance (AHA) - Active aging Â˚u Serious games Â˚u Assistive robotics Â˚u Augmented reality Â˚u Virtual coach