The Usability of an Exercise-based Tele-rehabilitation Service

A Hybrid Methodology

M. H. A. Huis in ’t Veld, S. M. Jansen-Kosterink

Roessingh Research and Development, Roessinghbleekweg 33b, Enschede, The Nederlands

S. Scattareggia, D. Wever

SignoMotus, via Panoramica dello Stretto, 340, Sicily, Italy

Roessingh Centre for Rehabilitation, Roessinghbleekweg 33, Enschede, The Netherlands

F. Benvenuti, W. Glinkowski

Azienda Unità Sanitaria Locale 11 di Empoli, Regione Toscana, Italy

Medical University of Warsaw, 61 Żwirki i Wigury Str. Warsaw, Poland

R. Sánchez-Carrión, S. Doménech

Fundació Privada Institut de Neurorehabilitació Guttmann. Camí de Can Ruti s/n, 08916 Badalona, Spain

Institut de l’Envelliment de la Universitat Autònoma de Barcelona

Sant Antoni Maria Claret 171. 08041 Barcelona, Spain

H. J. Hermens, M. M. R Vollenbroek-Hutten

Roessingh Research and Development, Roessinghsbleekweg 33b, Enschede, The Nederlands

University of Twente, Faculty of Electriclal Engineering, Mathematics and computer Science, Enschede, The Netherlands

Keywords: Usability, Exercise-based Tele-rehabilitation, Problem Software Reports, Change Requests, Methodology.

Abstract: Having usable technology is an imperative, especially for contemporary (elderly) patients with disabilities.

The aim of the present document is to examine the usability of our exercise-based tele-rehabilitation service

based on a hybrid approach in four different impaired user groups (elderly people with cognitive and/or

movement and postural problems) and in its context (as a tele-rehabilitation service during rehabilitation

treatment). A hybrid approach consisted of a task-oriented patient usability test and a one-year pilot-

implementation during which problem software reports and change requests were collected from

professionals. Across Europe, in total n=60 patients were included in the usability test suffering from

pulmonary diseases, stroke, orthopeadic patients and patients suffering from Alzheimer. In addition, during

the one-year pilot-implementation period in four European clinical centers (NL, PL, IT, ES), in total n=81

Software Problem Reports (SPR’s) have been collected and n=43 Change Requests (CRs). In conclusion,

this hybrid methodology allowed usability data to be retrieved from both an episodic and a longer period of

use, controlled use and use in routine care, and focus on both the tele-rehabilitation software and service

delivery, i.e tele-treatment protocols. Moreover, both the patient as well the professional perspective was

incorporated.

Sánchez-Carrión R., Jansen-Kosterink S., Huis in ’t Veld R., Doménech S., Wever D., Vollenbroek M., Benvenuti F., Scattareggia S., Glinkowski W. and

Hermens H. (2012).

The Usability of an Exercise-based Tele-rehabilitation ServiceA Hybrid Methodology.

In Proceedings of the Sixth International Symposium on e-Health Services and Technologies and the Third International Conference on Green IT

Solutions, pages 36-43

DOI: 10.5220/0004473800360043

 SciTePress

1 INTRODUCTION

The ageing of the population which leads to an

aggravation of chronic conditions, the growing need

for patients to become actors in their own health,

and the necessity of controlling health care whilst

maintaining high quality of care, place challenging

demands on our current health care system. Within

the European CLEAR project (= Clinical Leading

Environment for the Assessment of Rehabilitation

protocols in home care), an exercise based tele-

rehabilitation service and clinical protocols have

been developed and pilot-implemented in four

clinical centers across Europe to treat (elderly)

patients affected by pulmonary disease, stroke,

Alzheimer, and orthopedic problems (requiring often

total joint replacement) (www.habiliseurope.com).

Having usable technology is an imperative,

especially for contemporary (elderly) patients with

disabilities (Alexander and Staggers, 2009). The

common goals of usability research are to improve

the effectiveness, efficiency and satisfaction of

people interacting with tools (Hornbæk 2006).

Effectiveness includes the usefulness of a tool to

complete (work) tasks. Examples of efficiency

include time to complete specific tasks, the number

of clicks to perform tasks or the amount of time

needed. Satisfaction can include the perception of

any aspect of the tool typically includes perceptions

of the workload or the effectiveness of the specific

design (Alexander and Staggers 2009).

Concerning the ISO 9241 definition, usability

deals with the extent to which a product can be used

by specified users to achieve the before mentioned

specified goals (effectiveness, efficiency, and

satisfaction) in a specified context. The users, goals

(tasks) and context of our exercise-based tele-

rehabilitation program are defined in Table 1.

Table 1: Specified users, goals and context.

Users Patients suffering from pulmonary diseases,

Alzheimer, Stroke and Arthritis

Rehabilitation physicians, psychologists,

physiotherapists

Goals To enable multiple patients to exercise/train

simultaneously in their own relevant context

(either at home or at the clinic) under the

remote supervision of an expert professional

Context The location where the tele-rehabilitation is

provided: at home, at the clinic or outpatient

treatment facility

The country where the tele-rehabilitation

service is provided: The Netherlands,

Poland, Italy and Spain

How to measure usability, by which tests and

which parameters to be selected are important

challenges in human-computer-interaction research

(ISO, 1998; Frøkjær, 2000). The innovative and new

context of tele-rehabilitation services even more

increases the challenges associated with usability

examination because it involves new technologies,

new protocols for health care delivery (i.e. at home)

and (often) impaired user groups (Monk, 2002).

Expanding the knowledge on usability issues in the

field of tele-rehabilitation would enable better fit

between users (patients and professionals) and

technology, thereby possibly maximizing the

chances of successful adoption of tele-rehabilitation

services in routine care.

The aim of the present document is to examine

the usability of our exercise-based tele-rehabilitation

service based on a hybrid approach in four different

impaired user groups. This methodology is aimed at

evaluating the usability of the service in its context

(as a tele-rehabilitation service during treatment),

thus the usability of the protocol for home-based

treatment and keeping in mind a specific impaired

target group (elderly people with cognitive and/or

movement and postural problems).

2 METHOD

2.1 Exercise Tele-rehabilitation Service

The exercise tele-rehabilitation service consists of a

notebook with webcam, with newly developed

software giving access to a database of exercise

videos and a teleconference service to facilitate

contact between the patient and professional. With

this new service the professional can compose a

tailored exercise program for his patient. The patient

can carry out the program on a self-scheduled time

at the clinical at home or. The patient can record an

exercise with the webcam and the recorded exercise

can be assessed by the professional. Patient and

professional can contact each other by

teleconference to discuss the rehabilitation progress.

The professional can schedule, add and delete

exercises in the exercise program of the patient

during the rehabilitation. The exercise tele-

rehabilitation service enables patients to exercise at

home at moments preferred by the patients, which

fits in the current trend of self management of the

patient. Screenshots of the patient GUI is presented

in Appendix 1.

The Usability of an Exercise-based Tele-rehabilitation Service - A Hybrid Methodology

Table 2: Patient usability test: evaluation parameters based on framework of Hornbæk (2006).

2.2 Methodology

A hybrid methodology was used in evaluating the

usability of the exercise-based tele-rehabilitation

service.

(1) Patient usability test: consisting of a task-ori-

ented examination of the usability during episodic

use of the program in a laboratory setting

(2) Pilot-implementation: collecting problem soft-

ware reports and change requests during a one-year

period among professionals in order to provide

information on usability issues during routine use in

rehabilitation setting.

2.2.1 Patient Usability Test

Four different patient populations were invited to

participate in the usability study namely patients

with pulmonary diseases (The Netherlands), stroke

(Italy), orthopeadic patients (Poland) and patients

suffering from Alzheimer (Spain). All patients were

invited to local institutes to participate in a 2-hour

usability test during which they have to complete

four pre-defined concrete tasks being (1) start your

computer and log in, (2) start the patient exercise

module and watch tutorial, (3) Execute an exercise

and record the execution with video, (4)

communicate with the therapist by

videoconferencing. The usability parameters defined

by Horbæk (2006) were operationalized into

concrete parameters for this study (see Table 2).

2.2.2 Pilot-implementation Test

The exercise-based tele-rehabilitation program was

pilot-implemented as part of the traditional care

service for a period of one year. Besides

investigating the clinical impact, the scope of this

pilot-implementation was to collect in systematic

way relevant information regarding the requested

improvements of the service as a result of actual use

in a daily setting for a long(er) period of time.

The feedback collected have been divided in two

main categories (a) Software Problem Report (SPR)

and (b) Change Requests (CR). SPR’s deal with

information related to software malfunctions of the

service that prevent the proper delivery of the

service. These are mainly related to bad functioning

of modules noticed by the health care professionals

during their activities with the software. CRs deal

with a request of modification that is related to the

particular service model in use. These requests deals

EHST/ICGREEN 2012

Table 3a: Patient usability test results for effectiveness and efficiency.

mainly with improvements request of a particular

module or function that is not optimal for the scope,

addition of new functionalities, interoperability

aspects related to integration with the Hospital

Information System (HIS) in use in the concerned

centre.

2.3 Statistical Analyses

All analysis will be conducted on a group-level. For

the usability parameters containing ordinal

answering scales derived from the patient test, non-

parametric descriptive statistics (median and range)

will be calculated. For the usability parameters

containing a dichotomous answering format,

percentages will be calculated. For the pilot-

implementation the amount of SPRs and CRs will be

presented as well as a summative description of the

categories for which SPRs and CRs will be

provided.

3 RESULTS

3.1 Patient Usability Test

In total n=60 patients participated in the usability

test. Among these are n=24 males and n=36 females.

The mean age of the participants is 63.6 years (range

31-94). The results of the patient usability test are

presented per parameter per task per pathology in

Table 3a.

3.1.1 Effectiveness

Except for the patients suffering from cognitive

impairments due to Alzheimer’s disease, participants

were confident about their ability to remember how

to execute the different tasks. Interestingly, on

average few mistakes were made in the completion

of the tasks by the participants. However, the ranges

indicate that mistakes were made. Typically,

mistakes were associated with remembering

passwords or typing errors (task 1), forgetting to

click the start (Task 2, 3) and/or stop button (Task

3). In addition, as illustrated by the percentage of

patients who not completed the tasks successfully

the tele-rehabilitation software also contained some

problems. For instance, webcam connectivity

problems were faced as well as poor image quality

during videoconferencing (Task 4). As indicated by

the discrepancy in % interpretation of successful

completion between the examinator and

patients’self-report, Alzheimer patients’ seem to be

not fully aware whether or not they completed the

tasks successfully.

3.1.2 Efficiency

As shown in Table 3a, the duration to log in (Task 1)

and to start and watch the exercise tutorial (Task 2)

was rather comparable between patients suffering

from pulmonary diseases, stroke and orthopedic

problems. Patient suffering from Alzheimer’s

disease needed more time to log in. For task 3, large

differences existed between the patient populations.

The Usability of an Exercise-based Tele-rehabilitation Service - A Hybrid Methodology

Table 3b: Results of the laboratory usability for satisfaction of use (n=60).

These data can however not be directly compared

between the populations since they all recorded

different exercises, relevant to their pathologies. The

relatively long duration to set up a

videoconferencing by patients with pulmonary

diseases could be explained by connectivity

problems (one patient had to wait three minutes).

One patient forgot to put on the headset which was

necessary to communicate by voice/audio with the

therapist. Remarkably, despite the fact that patients

were rather satisfied about the ease of use (>3.5) of

the tele-rehabilitation program none of the patient

groups was able to complete the tasks without help

(to a greater or lesser extent). As could be expected,

patients suffering from Alzheimer had an inefficient

use pattern of the software which is (again) probably

due to their cognitive impairments

3.1.3 Satisfaction

The results of the satisfaction parameters are

illustrated in Table 3b for the different pathologies.

On average, patients were satisfied with the text

length and understandability of the instructions as

well as the quality of the navigation, text size, colors

and pictograms (median satisfaction scores > 3.0).

The quality of the interaction with the software was

perceived sufficient to excellent varying from grade

6.0 by patients suffering from pulmonary diseases to

a score of 9.0 by patients suffering from orthopedic

problems. In general, the majority of patients (>

69.2%) patients stated that they would recommend

the software to another patient or friend.

However, the variance in percentage of patients

who preferred tele-rehabilitation over traditional

care is high between the pathologies. In our opinion,

this variance could be explained by the difference in

treatment protocols, i.e. the extent to which the tele-

rehabilitation service substitutes traditional care.

For instance, 80% of the orthopedic patients prefer

tele-rehabilitation versus 15.4% of the pulmonary

patients. In the latter group, tele-rehabilitation

partially substituted traditional care whereas in the

orthopedic group tele-rehabilitation was provided

supplementary to traditional care.

3.2 Pilot-implementation Test

During a one-year pilot-implementation period in

four European clinical centers (NL, PL, IT, ES), in

EHST/ICGREEN 2012

total n=81 Software Problem Reports (SPR’s) have

been collected and n=43 Change Requests (CRs).

The exact nature of the SPR’s and CRs are too

specific for the technology used to describe and

analyze in detail in the current paper.

In general, the SPRs sent by the clinical centers

concern hanging applets, usability of log in and

tutorials for patients and professionals, bugs, more

flexible use of the tutorial management (should be

possible to add new exercises or remove ones and

show adequate sequence of exercises), connectivity

and/or audio quality of the videoconferencing,

patient exercise interface, webcam management, and

browser interactions.

In general, the description of the CRs concern

viewing back a just recorded exercise, delete a just

recorded exercise and replace by a new one, add

mandatory fields in forms, network policies, to allow

therapists to define a proper order in which the

selected tutorial will be shown to the patient,

increase the size of the log out button, bandwidth

issues, a videoconferencing call should not be able

to interrupt the patient during exercising, translation

of content to different languages, add personalized

exercise instructions, and client stations should be

allowed to access the service without the needs of a

VPN connection.

4 CONCLUSIONS

The current study investigated the usability of an

exercise-based tele-rehabilitation service in four

different countries, representing four different

patient populations being patients suffering from

pulmonary diseases, stroke, orthopeadic problems or

Alzheimer’s disease. Patients (n=60) were

characterized by physical and/or cognitive

impairments typically associated with their

pathologies and relatively high age. A hybrid

usability methodology was embraced consisting of

(1) a laboratory patient usability test and (2) a pilot-

implementation period of one year duration.

For the usability test, the patients had to execute

four predefined tasks (start up PC and log in to the

service, start and view exercise tutorial, record an

exercise, start and conduct a videoconferencing with

a therapist). The usability parameters examined were

derived from the framework of Hornbæk (2006).

During the one-year pilot implementation the

usability of the service was examined by means of

collecting n=81 Software Problem Reports (SPRs)

and n=43 Change Requests (CR).

The innovativeness and strength of our hybrid

methodology lies in the retrieval of usability data

from both an episodic and a longer period of use,

controlled use and use in routine care, and focus on

the tele-rehabilitation software and service delivery,

e.g. tele-treatment protocols. Moreover, both the

patient as well the professional perspective was

incorporated in this methodology. In addition, our

operationalization of the framework of Hornbæk

(2006) into concrete parameters could be valuable

for other researchers in the field of evaluating the

usability of similar services.

Because of the fact that the patient study and

pilot-implementation contained different

methodologies and parameterizations not all results

may be easily comparable. In our view, this should

not be considered to be a weakness of the

methodology but a valuable strength. Together this

information provides a broad(er) perspective on

usability and recommendations to improve it.

Interestingly, commonalities in usability issues were

found between the patient usability test and the pilot

implementation. These commonalities dealt with

to the videoconferencing module and connectivity.

The methodology could be improved by

repeating the patient usability test over time during

the (pilot-) implementation (Hornbæk, 2006).

Another recommendation for methodology

improvement is to extend the measure of satisfaction

by post-use Likert-scales as done in the current

study by validated and standardized questions. As

illustrated by the ‘regression to the mean’ trend,

results of the satisfaction questionnaire tended

towards ‘socially desirable’ answers by the users.

Lastly, the parameters used are not intended to

serve as an exclusive list of usability parameters.

Instead, according to Hornbæk (2006), it provides

usability researchers an overview of the possible

parameters that can be applied. Researchers should

select the parameters to be most relevant for their

own research purposes and concretized for the

service they want to evaluate.

As could be expected on advance, patients with

Alzheimer’s disease had more difficulty in using the

tele-rehabilitation service compared to patients

suffering from pulmonary diseases, orthopaedic

problems and stroke. This is probably due to their

severe cognitive impairments of the Alzheimer

population included in the present study. This result

does reveal the recommendation for other

investigators to take into account the characteristics

of the target users and their pathologies when

designing innovative technology-based treatment

The Usability of an Exercise-based Tele-rehabilitation Service - A Hybrid Methodology

concepts. For instance, instead of remembering

passwords fingerprints log in would have been more

suited for patients with cognitive impairments.

Results showed that the majority of patients were

able to complete the tasks successfully without many

unnecessary actions and mistakes. However, despite

the fact that all patients were trained in using the

service under study prior to participation (some of

them actually used the service during their rehab

treatment!) and that patients reported to be rather

satisfied about its ease of use, the majority of them

still needed help/assistance. Apparently, it is not

realistic to assume that patients who passed the

training and believe they are confident to use the

service are actually able to use the service

adequately independently. This result has major

clinical implications. Without having insight into the

effectiveness and efficiency of use of software by

patients, therapists will hardly be able to explain

(poor) clinical impact due to inadequate use and the

(content of the) treatment. This lack of insight in the

use can be addressed by the technology and/or the

service delivery. From a technology perspective,

automatic usage logs and compliance measures

could be integrated in the system. From a service

delivery perspective, it is strongly recommended to

organize assistance for patients while interacting

with the service e.g. by therapists (when interacting

inside the rehab facility) or informal caregivers

(when interacting at home).

Results showed that the majority of patients

would recommend the service to other patients or

friends. Major reasons for recommending the service

is the fact that it offers the opportunity to receive

additional specific instructions during training (at

home or at the rehab facility) as a mnemonic.

Nevertheless, the patients groups differed in the

degree to which they preferred the tele-rehabilitation

service compared to regular in-person care (with

lowest preference score for patients with pulmonary

diseases and highest score for patient with

Alzheimer disease). These differences could be

explained by the differences in the way the tele-

rehabilitation is embedded in regular in-person care

during the pilot-implementation between the groups.

For instance, for pulmonary patients, tele-

rehabilitation is offered as a (partly) replacement of

traditional in-person care, i.e. service delivery

whereas for patients with Alzheimers’ disease the

tele-rehabilitation service is used as an extra training

modality inside the rehabilitation facility.

Consequently by carefully considering an optimal

design for the service delivery for tele-rehabilitation

innovations one could be able to affect the users’

attitude and possibly the degree of (future) adoption

by these users.

A recent systematic review showed that the

usability evaluations differed in methodologies and

quality indicators for usability making it difficult to

reveal abstractions, i.e. commonalities in failures

and facilitators for usability of clinical technology

that can be used by others (Alexander and Staggers,

2009). However, it can be concluded that the present

study revealed at least three recommendations for

usability design for exercise-based tele-rehabilitation

services. First, one should carefully consider the

characteristics (e.g. cognitive versus physical

impairments) of the pathologies of users involved.

Our results showed different usability issues for the

different pathologies. Second, insight in adequate

use of the technology is crucial when evaluating the

clinical impact. Our results showed that the majority

patients were not capable of using the program

adequately despite them having successfully passed

their instruction session. Third, careful consideration

of how the tele-rehabilitation is embedded in regular

in-person care, i.e. the service delivery, affects the

attitude of patients toward and possibly the

successful adoption of these innovative

interventions.

REFERENCES

Habilis Europe http://www.habiliseurope.eu/

Hornbæk K 2006, Current practice in measuring usability:

challenges to usability studies and research,

International Journal of Human-Computer Studies,

64, 79-102

Alexander G and Staggers N (2009), A systematic review

on the designs of clinical technology: findings and

recommendations for future research, ANS Adv Nurs

Scie, 32 (3), 252-279

ISO, 1998, Ergonomic requirements for office works with

visual display terminals (VDTs) – Part 11 guidance

on usability (ISO 9241-11)

Frøkjær E, Hertzum M, Hornbæk, 2000, Measuring

usability: are effectiveness, efficiency, and satisfaction

really correlated. In: Proceedings of ACM Conference

on Human Factors in Computer Systems, ACM Press,

New York, 354-352

Monk A 2002. Noddy’s guide to usability. Interfaces 50,

31–33

APPENDIX 1

Screenshots of the tele-rehabilitation service

EHST/ICGREEN 2012

Chat

Hospital

Logo

onnect

ote video:

Disconnect

ideoconferenc

ssigning exer

Example

Local video:

Record

ng therapist -

ise from librar

of an exercise

Send

Home | Print | Help

Frame Rate Quality

rofessional

to patient

ideo

The Usability of an Exercise-based Tele-rehabilitation Service - A Hybrid Methodology