A Scoping Review of the Inquiry Instruments Being Used to Evaluate

the Usability of Ambient Assisted Living Solutions

Rute Bastardo

, João Pavão

and Nelson Pacheco Rocha

UNIDCOM, Science and Technology School, University of Trás-os-Montes and Alto Douro, Vila Real, Portugal

INESC-TEC, Science and Technology School, University of Trás-os-Montes and Alto Douro, Vila Real, Portugal

IEETA-Institute of Electronics and Informatics Engineering of Aveiro, Department of Medical Sciences,

University of Aveiro, Aveiro, Portugal

Keywords: Usability Evaluation, Inquiry Instruments, Ambient Assisted Living, Older Adults, Scoping Review.

Abstract: This paper reports a scoping review of the literature to identify the inquiry instruments being used to evaluate

the usability of AAL solutions, which resulted in the inclusion of 35 studies. The results show that a significant

number of the included studies reported the use of non-valid inquiry instruments, such as ad-hoc

questionnaires. Among the studies using valid and reliable inquiry instruments, System Usability Scale (SUS)

emerged as the most used one. In general, valid and reliable inquiry instruments are being used together with

additional data gathering methods, to perform comprehensive usability evaluations. Moreover, in terms of the

quality of the design of the included studies, it should be pointed the adequacy of the participants’

characteristics and the tasks they performed. In turn, these studies did not present evidence of the preparation

and independence of the evaluators.

1 INTRODUCTION

Ambient Assisted Living (AAL) is one of the

resources available to support the increasing older

adults’ population, not only to optimize healthcare

services but also to mitigate the individuals’

disabilities. It refers to intelligent technologies,

products and services embedded in the physical

environment to support the care of older adults and to

promote their autonomy, independence, safety, social

participation, and well-being (Queirós et al., 2015).

A distinctive characteristic of AAL is the

interaction with all kinds of elements through user

interfaces that comprise multimodal interactions,

including modalities such as, for instance voice,

haptic, gesture or body movement interaction. User

satisfaction and performance are key aspects of these

interfaces based on novel concepts and control

schemes together with context awareness. High

complexity in terms of implementation of user

interaction mechanisms must be translated in simple

and usable interfaces.

Therefore, usability evaluation is essential to

surpass design problems and to guarantee that ALL

https://orcid.org/0000-0003-3801-7249

solutions might be accepted and used by their target

users (i.e., older adults).

Over real systems or prototypes, the best

alternatives are either evaluations conducted by

experts, which are also known as inspection (da Costa

et al., 2019; Dix et al., 2004), or evaluations involving

users that might use testing and inquiry methods

(Bernsen & Dybkjær, 2010). These methods can be

combined to perform comprehensive usability

evaluations (Martins, Queirós, et al., 2015).

Usability testing (e.g., observation or

performance evaluation) usually involves observing

users and measuring their performance while they

perform predefined tasks. The respective methods

focus on the users and their tasks, and seek empirical

evidence, mostly quantitative, about how to improve

the user interaction (Martin & Hanington, 2012).

In turn, inquiry methods (e.g., interviews, focus

groups, or questionnaires) involve collecting the

perceptions of the users. Although the data collected

are subjective, inquiry methods provide valuable

information on what the users want and help to

identify usability strengths and weaknesses (Martins,

Queirós, et al., 2015).

320

Bastardo, R., Pavão, J. and Rocha, N.

A Scoping Review of the Inquiry Instruments Being Used to Evaluate the Usability of Ambient Assisted Living Solutions.

DOI: 10.5220/0010766600003123

In Proceedings of the 15th International Joint Conference on Biomedical Engineering Systems and Technologies (BIOSTEC 2022) - Volume 5: HEALTHINF, pages 320-327

ISBN: 978-989-758-552-4; ISSN: 2184-4305

Established instruments such as the System

Usability Scale (SUS) (Brooke, 1996; Martins et al.,

2015), Post-Study System Usability Questionnaire

(PSSUQ) (Lewis, 1992) or International

Classification of Functioning, Disability and Health

Usability Scale (ICF-US) (Martins, Rosa, et al.,

2015), are inquiry instrument being used for the

evaluation of usability of a wide range of products

and user interfaces.

This scoping review aimed to identify the inquiry

instruments being used to evaluate the usability of

AAL solutions. The study intends to contribute to the

quality of user-centred usability evaluation of AAL

solutions by reviewing the main research recently

published and determining and discussing the

methods and inquiry instruments being used.

The rest of the paper is organized as follows: the

next section presents the research questions and

methods of the present scoping review; the results are

presented in Section 3; Section 4 provides the

discussion of the results and a conclusion.

2 RESEARCH QUESTIONS AND

METHODS

Considering the aforementioned objective, the

scoping literature review was informed by the

following research questions:

 RQ1. What are the inquiry instruments being

used to evaluate the usability of ALL solutions?

 RQ2. Are the reported inquiry instruments

valid and reliable?

 RQ3. What is the quality of the experimental

design of the studies using valid and reliable

inquiry instruments?

The authors defined a protocol to perform this

scoping literature with explicit descriptions of the

methods to be used and the steps to be taken, which

is described in this section.

The resources chosen for the review were three

electronic databases (i.e., Scopus, Web of Science,

and IEEE Xplorer). Boolean queries were prepared to

include all the articles that have their titles, abstract

or keywords conformed with the conjunction (i.e.,

AND Boolean operator) of the following Boolean

expressions:

 ‘AAL’ OR ‘Ambient assisted living’ OR

‘ambient assisted technology’ OR ‘ambient

assistive technology’ OR ‘ambient

intelligence’.

 ‘UX’ OR ‘user experience’ OR ‘usability’.

 ‘Evaluation’ OR ‘assessment’.

The electronic literature search was performed in

January 2021 and included all the references

published before December 31, 2020.

The inclusion and exclusion criteria were the

following:

 References were included if they reported

usability evaluation of AAL solutions based on

inquiry methods.

 References were excluded if they: i) did not

have abstracts; ii) were not written in English;

iii) reported on reviews, surveys, or market

studies; iv) were books, reported workshops, or

were special issues announcements; and v)

reported on studies that are not relevant for the

objective of this scoping review.

The analysis and selection of the references were

performed in three steps:

 First step - the authors removed the duplicates,

the references without abstract and that were

not written in English.

 Second step - the authors assessed all titles and

abstracts for relevance and those clearly not

meeting the inclusion and exclusion criteria

were removed.

 Third step - the authors then assessed the full

text of the remaining references against the

outlined inclusion and exclusion criteria and

the final list of the studies to be included for the

review was created.

Throughout this entire process, all the references

were analysed by three authors and any disagreement

between the authors was discussed and resolved by

consensus.

The analysis of the included studies considered

the following dimensions: i) demographic

characteristics; ii) purposes; iii) interaction

modalities; iv) usability evaluation methods; v)

usability inquiry instruments; and vi) quality of the

experimental design.

Based on the demographic data, a synthesis of

studies’ characteristics was prepared, which included:

i) the number of studies published in conference

proceedings and in scientific journals; and ii) the

distribution of the studies by year.

Additionally, tabular presentations were prepared

with the purposes of the AAL solutions being

reported and the respective interaction modalities

(i.e., visual interaction together with voice, auditory,

gesture or other interaction modalities, such as

immersive virtual reality or robots).

Moreover, the included studies were analysed and

classified according to the type of the inquiry

instruments being used.

A Scoping Review of the Inquiry Instruments Being Used to Evaluate the Usability of Ambient Assisted Living Solutions

321

Finally, concerning the subset of studies that used

valid and reliable inquiry instruments, an analysis of

the quality of their experimental design was

performed, considering the following aspects (Silva

et al., 2019): i) additional data gathering to allow the

triangulation of the usability evaluation results; ii)

adequacy of the number and characteristics of the

individuals participating in the usability studies; and

iii) usability evaluators.

3 RESULTS

3.1 Study Selection

A total of 5635 studies were retrieved from the search

of the selected databases.

The first step of the selection process yielded 2996

studies since 2639 studies were removed because they

were duplicated, did not have abstracts, or were not

written in English.

During the second step, 2919 studies were

removed, because they reported on reviews, surveys,

or market studies and workshops, were special issues

announcements, or were not relevant for the objective

of this scoping review.

Finally, after the full text analysis (i.e., the third

step) 42 studies were removed since they did not meet

the inclusion and exclusion criteria.

Therefore, 35 studies were included in this

scoping review (Table 1).

3.2 Demographic Characteristics

Looking to the types of publications, six articles

(Bleser et al., 2013; Cavallo, Aquilano, et al., 2014;

Chartomatsidis & Goumopoulos, 2019; Dias et al.,

2015; Werner et al., 2012; Zhunio et al., 2020) were

published in conference proceedings and the

remainder 29 articles were published in scientific

journals.

In turn, almost 75% of the studies were published

in the last five years and 40% of the articles were

published in the last two years.

3.3 Purposes of the Studies and

Interaction Modalities

As can be seen in Table 2, 11 different purposes

emerged from the analysis of the included articles: i)

daily living activities; ii) falls prevention; iii) home

monitoring; iv) remote care; v) telerehabilitation; vi)

medication management; vii) physical activity; viii)

cognitive activity; ix) physical and cognitive activity;

x) social inclusion; and xi) participation in leisure

activities.

All the studies reported the use of traditional

interfaces based in visual interaction. Moreover, as

represented in Table 3, 85% of the studies (i.e., 29

studies) reported additional interaction modalities,

namely voice, auditory, gesture and immersive virtual

reality interactions, or interaction with robots.

3.4 Inquiry Instruments

Table 4 presents the different types of inquiry

instruments reported by the included studies. In 12

studies, the usability evaluation was based on ad-hoc

questionnaires prepared by the researchers. In

general, the data gathered with these ad-hoc

questionnaires were complemented with other

evaluation techniques, namely interviews (Bleser et

al., 2013; Palestra et al., 2019), think aloud (Werner

et al., 2012), observation (Blasco et al., 2014), and

performance evaluation (Brauner & Ziefle, 2020;

Orso et al., 2017). Moreover, four studies considered

questionnaires based on acceptance models (i.e.,

Technology Acceptance Model - TAM and Unified

Theory of Acceptance and Use of Technology -

UTAUT), which in some cases were complemented

by interviews and performance evaluation

(Goumopoulos et al., 2017), or observation (Morán et

al., 2015).

Moreover, 18 studies (i.e., almost 60% of the

included studies) used valid and reliable inquiry

instruments: i) SUS (14 studies); ii) SUS and PSSUQ

(one study); iii) SUS and Computer System Usability

Questionnaire (CSUQ) (one study); and iv) ICF-US

(two studies).

3.5 Quality Assessment

Except four studies (Cortellessa et al., 2018;

Konstantinidis et al., 2016; Macis et al., 2019;

Wohlfahrt-Laymann et al., 2019), the remainder

studies using valid and reliable inquiry instruments

complement the results of these instruments with

additional data gathering methods, such as

interviews, observation, performance evaluation,

think aloud, and additional questionnaires, to

consolidate the usability evaluation by applying

triangulation techniques (Table 5).

Table 6 presents the results of the analysis of the

adequacy of the evaluator’s preparation and the

characteristics of the participants and the tasks they

performed. Only three studies referred the

characteristics of the evaluators, namely if they were

HEALTHINF 2022 - 15th International Conference on Health Informatics

322

adequately trained or if they were external to the

development of the AAL solution. Moreover, in 11

studies, the number and characteristics of the

participants were adequate. Finally, in all included

studies, the tasks that the participants needed to

perform were coherent with the functions of the AAL

solution being developed.

Table 1: Included studies.

References

(Adcock et al., 2020; Adcock et al., 2019; Blasco et al., 2014; Bleser et al., 2013; Brauner & Ziefle, 2020; Cavallo,

Aquilano, et al., 2014; Cavallo et al., 2018; Cavallo, Limosani, et al., 2014; Chartomatsidis & Goumopoulos, 2019;

Cortellessa et al., 2018; Costa et al., 2017; Costa et al., 2015; Delmastro et al., 2019; Di Nuovo et al., 2018; Dias et al.,

2015; Fiorini et al., 2017; Goumopoulos et al., 2017; Gullà et al., 2019; Konstantinidis et al., 2016; Macis et al., 2019;

Money et al., 2019; Morán et al., 2015; Orso et al., 2017; Palestra et al., 2019; Pedroli et al., 2018; Pripfl et al., 2016;

Rebsamen et al., 2019; Sánchez-Morillo et al., 2015; Teixeira et al., 2017; Vaziri et al., 2016; Werner et al., 2012;

Wohlfahrt-Laymann et al., 2019; Yilmaz, 2019; Zhunio et al., 2020; Zlatintsi et al., 2020)

Table 2: Purposes of the included studies.

Purposes References

Daily living activities

(Blasco et al., 2014; Cavallo, Aquilano, et al., 2014;

Cavallo et al., 2018; Cavallo, Limosani, et al., 2014; Di

Nuovo et al., 2018; Dias et al., 2015; Gullà et al., 2019;

Werner et al., 2012; Yilmaz, 2019; Zlatintsi et al., 2020)

Falls prevention

(Money et al., 2019; Pripfl et al., 2016; Vaziri et al.,

2016)

Home monitoring

(Costa et al., 2015; Delmastro et al., 2019; Macis et al.,

2019; Sánchez-Morillo et al., 2015; Wohlfahrt-Laymann

et al., 2019)

Remote care

(Cortellessa et al., 2018; Costa et al., 2017; Fiorini et al.,

2017)

Telerehabilitation

(Morán et al., 2015; Palestra et al., 2019; Pedroli et al.,

2018)

Medication management (Teixeira et al., 2017)

Physical activity

(Bleser et al., 2013; Brauner & Ziefle, 2020;

Chartomatsidis & Goumopoulos, 2019; Konstantinidis et

al., 2016; Rebsamen et al., 2019)

Cognitive activity (Zhunio et al., 2020)

Physical and cognitive activity (Adcock et al., 2020; Adcock et al., 2019)

Social inclusion (Goumopoulos et al., 2017)

Participation in leisure activities (Orso et al., 2017)

Table 3: Interaction modalities.

Interaction References

Auditory and voice interaction (Blasco et al., 2014; Bleser et al., 2013; Costa et al., 2017;

Dias et al., 2015; Goumopoulos et al., 2017; Macis et al.,

2019; Orso et al., 2017; Sánchez-Morillo et al., 2015;

Teixeira et al., 2017; Wohlfahrt-Laymann et al., 2019)

Gesture recognition (Adcock et al., 2020; Adcock et al., 2019; Brauner &

Ziefle, 2020; Chartomatsidis & Goumopoulos, 2019;

Morán et al., 2015; Palestra et al., 2019; Rebsamen et al.,

2019; Vaziri et al., 2016)

Immersive virtual reality (Gullà et al., 2019; Pedroli et al., 2018)

Robotics interaction (Cavallo, Aquilano, et al., 2014; Cavallo et al., 2018;

Cavallo, Limosani, et al., 2014; Cortellessa et al., 2018;

Di Nuovo et al., 2018; Fiorini et al., 2017; Pripfl et al.,

2016; Werner et al., 2012; Zlatintsi et al., 2020)

A Scoping Review of the Inquiry Instruments Being Used to Evaluate the Usability of Ambient Assisted Living Solutions

323

Table 4: Inquiry instruments.

Instruments References

Ad-hoc questionnaires (Blasco et al., 2014; Bleser et al., 2013; Brauner & Ziefle,

2020; Cavallo, Aquilano, et al., 2014; Cavallo, Limosani,

et al., 2014; Costa et al., 2017; Costa et al., 2015; Fiorini

et al., 2017; Orso et al., 2017; Palestra et al., 2019; Werner

et al., 2012; Yilmaz, 2019)

Questionnaires based on acceptance models

TAM (Goumopoulos et al., 2017; Morán et al., 2015; Zhunio et

al., 2020)

UTAUT (Cavallo et al., 2018)

Usability scales and questionnaires

ICF-US (Dias et al., 2015; Teixeira et al., 2017)

SUS (Adcock et al., 2020; Adcock et al., 2019; Chartomatsidis

& Goumopoulos, 2019; Delmastro et al., 2019; Di Nuovo

et al., 2018; Gullà et al., 2019; Konstantinidis et al., 2016;

Money et al., 2019; Pedroli et al., 2018; Rebsamen et al.,

2019; Sánchez-Morillo et al., 2015; Vaziri et al., 2016;

Wohlfahrt-Laymann et al., 2019; Zlatintsi et al., 2020)

SUS and PSSUQ (Macis et al., 2019)

SUS and CSUQ (Cortellessa et al., 2018)

Table 5: Additional data gathering in the studies using valid and reliable usability evaluation instruments.

Additional data gathering methods References

Interviews (Pedroli et al., 2018)

Interviews and observation (Vaziri et al., 2016)

Interviews and performance evaluation (Chartomatsidis & Goumopoulos, 2019)

Interviews, performance evaluation and think aloud (Money et al., 2019; Sánchez-Morillo et al., 2015)

Interviews and Mobile App Rating Scale (MARS) (Gullà et al., 2019)

Observation (Zlatintsi et al., 2020)

Observation and think aloud (Teixeira et al., 2017)

Observation and think aloud and Game Experience

Questionnaire (GEQ)

(Adcock et al., 2020)

Performance evaluation (Delmastro et al., 2019; Dias et al., 2015)

Performance and observation (Adcock et al., 2019)

Think aloud and TAM (Rebsamen et al., 2019)

UTAUT (Di Nuovo et al., 2018)

Table 6: Adequacy of the evaluator’s preparation, participants characteristics and tasks.

Evaluator’s preparation, participants characteristics

and tasks performed

References

Evaluators’ preparation (Adcock et al., 2019; Sánchez-Morillo et al., 2015)

Participants’ characteristics (Adcock et al., 2019; Di Nuovo et al., 2018; Gullà et al.,

2019; Macis et al., 2019; Money et al., 2019; Pedroli et

al., 2018; Rebsamen et al., 2019; Sánchez-Morillo et al.,

2015; Vaziri et al., 2016; Wohlfahrt-Laymann et al.,

2019; Zlatintsi et al., 2020)

Tasks (Adcock et al., 2020; Adcock et al., 2019; Chartomatsidis

& Goumopoulos, 2019; Delmastro et al., 2019; Di Nuovo

et al., 2018; Gullà et al., 2019; Konstantinidis et al.,

2016; Macis et al., 2019; Money et al., 2019; Pedroli et

al., 2018; Rebsamen et al., 2019; Sánchez-Morillo et al.,

2015; Vaziri et al., 2016; Wohlfahrt-Laymann et al.,

2019; Zlatintsi et al., 2020)

HEALTHINF 2022 - 15th International Conference on Health Informatics

324

4 DISCUSSION AND

CONCLUSION

Considering the selected databases, the search

queries, and the inclusion and exclusion criteria, this

scoping review identified 35 studies. In terms of the

purposes of the included studies, they are in line with

the current concerns related to the adoption of

technological solutions to support the increasing

older adults’ population, both in terms of care

provision and promotion of active ageing paradigms

(Jaschinski & Ben Allouch, 2019): i) healthcare

provision (i.e., home monitoring, remote care,

telerehabilitation and medication management); ii)

secure and supportive environment (i.e., daily living

activities and falls prevention); iii) healthy lifestyles

(i.e., physical activity and cognitive activity); and iv)

social involvement and active participation (i.e.,

social inclusion and participation in leisure activities).

The importance of usability evaluation increased

over the years and seems that the researchers are

looking beyond the technological perspective and are

interesting to gather the users’ opinions about new

AAL solutions. This is slightly different from the

results of (Queirós et al., 2015), which evidenced

focus on the development of technological solutions

rather than the development of services that could

satisfy the real needs of older adults.

The results also evidenced the involvement of

users in all the developmental phases, which is an

essential requirement when developing AAL

solutions (Queirós et al., 2017).

In terms of the first research question (i.e., what

are the inquiry instruments being used to evaluate the

usability of ALL solutions?), the results point to the

use of a diversity of instruments: i) ad-hoc

questionnaires - 12 studies; ii) questionnaires based

on acceptance models - four studies; iii) studies using

usability evaluation scales or questionnaires - 18

studies.

Concerning the validity of the reported inquiry

instruments (i.e., the second research question), ad-

hoc questionnaires might provide useful information

to assess design options, but they are not valid nor

reliable instruments to measure usability. This means

that their results are not reproducible neither

comparable. Moreover, acceptance models, such as

TAM or UTAUT, might be used to have a

comprehensive perspective of intentions of use, but

they are not adequate to discriminate usability

features.

In turn, established inquiry instruments were

designed to provide reliable and repeatable results, as

well as a depth understating of the usability features

being evaluated. Instruments such as SUS, PSSUQ,

ICF-US or CSUQ are valid and reliable inquiry

instruments to measure usability and to allow

comparability between different studies. These

instruments were applied in 18 of the includes studies.

Furthermore, SUS was the most relevant instrument

since it was used in 16 studies.

Considering the third research question (i.e., what

is the quality of the experimental design of the studies

using valid and reliable inquiry instruments?), it

should be noted that there is a concern in using

triangulation techniques, as it is recommended by the

literature (Silva et al., 2019). Fourteen of the 18

studies that report the use of valid and reliable inquiry

instruments also reported the use of additional data

gathering methods, such as interviews, observation,

performance evaluation or think aloud.

Moreover, in most studies the tasks performed by

the participants were representative of the functions

of the solutions being evaluated, as well as the

number and characteristics of the participants of the

experimental set-up were adequate.

In terms of negatives aspects, it should be pointed

that the included studies did not present evidence to

show that the usability evaluators were adequately

trained nor that they were external to the development

process. This aspect should be improved in future

studies.

The limitations of this scoping review are related

to the dependency on its keywords and the databases

selected. Despite these limitations, the authors

believe that the systematically collected evidence

contributes to the understanding of the current trends

of the development of AAL solutions. Therefore, it is

possible to conclude that future research related to

usability evaluation of AAL solutions should

consider using valid and reliable inquiry instruments

such as SUS, instead of ad-hoc questionnaires.

Moreover, special attention should be considered to

the preparation and independence of the evaluators

and the possibility of conducting field trials for long

periods in the real context or close to the real context

where the AAL solutions are going to be used.

REFERENCES

Adcock, M., Sonder, F., Schättin, A., Gennaro, F., & De

Bruin, E. D. (2020). A usability study of a

multicomponent video game-based training for older

adults. European Review of Aging and Physical

Activity, 17(1), 1-15.

Adcock, M., Thalmann, M., Schättin, A., Gennaro, F., & de

Bruin, E. D. (2019). A Pilot Study of an In-Home

Multicomponent Exergame Training for Older Adults:

A Scoping Review of the Inquiry Instruments Being Used to Evaluate the Usability of Ambient Assisted Living Solutions

325

Feasibility, Usability and Pre-Post Evaluation.

Frontiers in Aging Neuroscience, 11, 304.

Bernsen, N., & Dybkjær, L. (2010). Multimodal Usability.

https://doi.org/10.1007/978-1-84882-553-6 (Human-

Computer Interaction Series)

Blasco, R., Marco, Á., Casas, R., Cirujano, D., & Picking,

R. (2014). A Smart Kitchen for Ambient Assisted

Living. Sensors, 14(1), 1629-1653.

Bleser, G., Steffen, D., Weber, M., Hendeby, G., Stricker,

D., Fradet, L., Marin, F., Ville, N., & Carré, F. (2013).

A personalized exercise trainer for the elderly. Journal

of Ambient Intelligence and Smart Environments, 5(6),

547-562.

Brauner, P., & Ziefle, M. (2020). Serious motion-based

exercise games for older adults: Evaluation of usability,

performance, and pain mitigation. JMIR Serious

Games, 8(2).

Brooke, J. (1996). SUS: A 'Quick and Dirty' Usability

Scale. Usability Evaluation in Industry, 189, 4-7.

Cavallo, F., Aquilano, M., Bonaccorsi, M., Limosani, R.,

Manzi, A., Carrozza, M. C., & Dario, P. (2014).

Improving Domiciliary Robotic Services by Integrating

the ASTRO Robot in an AmI Infrastructure. Gearing

up and accelerating cross

‐

fertilization between

academic and industrial robotics research in Europe,

267-282.

Cavallo, F., Limosani, R., Fiorini, L., Esposito, R., Furferi,

R., Governi, L., & Carfagni, M. (2018). Design impact

of acceptability and dependability in assisted living

robotic applications. International Journal on

Interactive Design and Manufacturing, 12(4), 1167-

1178.

Cavallo, F., Limosani, R., Manzi, A., Bonaccorsi, M.,

Esposito, R., Rocco, M., Pecora, F., Teti, G., Saffiotti,

A., & Dario, P. (2014). Development of a Socially

Believable Multi-Robot Solution from Town to Home.

Cognitive Computation, 6, 954-967.

Chartomatsidis, M., & Goumopoulos, C. (2019). A balance

training game tool for seniors using Microsoft Kinect

and 3D Worlds. ICT4AWE 2019 - Proceedings of the

5th International Conference on Information and

Communication Technologies for Ageing Well and e-

Health, 135-145.

Cortellessa, G., Fracasso, F., Sorrentino, A., Orlandini, A.,

Bernardi, G., Coraci, L., De Benedictis, R., & Cesta, A.

(2018). ROBIN, a Telepresence Robot to Support Older

Users Monitoring and Social Inclusion: Development

and Evaluation. Telemedicine and e-Health, 24(2), 145-

154.

Costa, C. R., Anido-Rifon, L. E., & Fernandez-Iglesias, M.

J. (2017). An Open Architecture to Support Social and

Health Services in a Smart TV Environment. IEEE

Journal of Biomedical and Health Informatics, 21(2),

549-560.

Costa, S. E., Rodrigues, J. J., Silva, B. M., Isento, J. N., &

Corchado, J. M. (2015). Integration of Wearable

Solutions in AAL Environments with Mobility Support.

Journal of medical systems, 39(12), 1-8.

da Costa, R., Canedo, E., de Sousa, R., Albuquerque, R., &

Villalba, L. (2019). Set of Usability Heuristics for

Quality Assessment of Mobile Applications on

Smartphones. IEEE Access, 7, 116145-116161.

Delmastro, F., Dolciotti, C., La Rosa, D., Di Martino, F.,

Magrini, M., Coscetti, S., & Palumbo, F. (2019).

Experimenting mobile and e-health services with frail

MCI older people. Information (Switzerland), 10(8),

253.

Di Nuovo, A., Broz, F., Wang, N., Belpaeme, T., Cangelosi,

A., Jones, R., Esposito, R., Cavallo, F., & Dario, P.

(2018). The multi-modal interface of Robot-Era multi-

robot services tailored for the elderly. Intelligent

Service Robotics, 11(1), 109-126.

Dias, M. S., Vilar, E., Sousa, F., Vasconcelos, A., Pinto, F.

M., Saldanha, N., & Eloy, S. (2015). A Living Labs

Approach for Usability Testing of Ambient Assisted

Living Technologies. Marcus A. (eds) Design, User

Experience, and Usability: Design Discourse. Lecture

Notes in Computer Science, 9186, 167-178.

Dix, A., Finlay, J., Abowd, G., & Beale, R. (2004). Human-

computer interaction (3rd ed.). Pearson/Prentice-Hall.

Fiorini, L., Esposito, R., Bonaccorsi, M., Petrazzuolo, C.,

Saponara, F., Giannantonio, R., Petris, G. D., Dario, P.,

& Cavallo, F. (2017). Enabling personalised medical

support for chronic disease management through a

hybrid robot-cloud approach. Autonomous Robots,

41(5), 1263-1276.

Goumopoulos, C., Papa, I., & Stavrianos, A. (2017).

Development and Evaluation of a Mobile Application

Suite for Enhancing the Social Inclusion and Well-

Being of Seniors. Informatics, 4(3), 15.

Gullà, F., Menghi, R., Papetti, A., Carulli, M., Bordegoni,

M., Gaggioli, A., & Germani, M. (2019). Prototyping

adaptive systems in smart environments using virtual

reality. International Journal on Interactive Design and

Manufacturing, 13(2), 597-616.

Jaschinski, C., & Ben Allouch, S. (2019). Listening to the

ones who care: exploring the perceptions of informal

caregivers towards ambient assisted living applications.

Journal of Ambient Intelligence and Humanized

Computing, 10(2), 761-778.

Konstantinidis, E. I., Bamparopoulos, G., & Bamidis, P. D.

(2016). Moving Real Exergaming Engines on the Web:

The webFitForAll Case Study in an Active and Healthy

Ageing Living Lab Environment. IEEE Journal of

Biomedical and Health Informatics, 21(3), 859-866.

Lewis, J. (1992). Psychometric evaluation of the post-study

system usability questionnaire: The PSSUQ.

Proceedings of the Human Factors Society.

Proceedings of the Human Factors Society, 2, 1259-

1263.

Macis, S., Loi, D., Ulgheri, A., Pani, D., Solinas, G., La

Manna, S., Cestone, V., Guerri, D., & Raffo, L. (2019).

Design and Usability Assessment of a Multi-Device

SOA-Based Telecare Framework for the Elderly. IEEE

Journal of Biomedical and Health Informatics, 24(1),

268-279.

Martin, B., & Hanington, B. M. (2012). Universal methods

of design : 100 ways to research complex problems,

develop innovative ideas, and design effective

solutions. 207-207.

HEALTHINF 2022 - 15th International Conference on Health Informatics

326

Martins, A. I., Queirós, A., Silva, A. G., & Rocha, N. P.

(2015). Usability evaluation methods: A systematic

review. Human Factors in Software Development and

Design, 250-273.

Martins, A. I., Rosa, A. F., Queirós, A., Silva, A., & Rocha,

N. P. (2015). Definition and Validation of the ICF –

Usability Scale. Procedia Computer Science, 67, 132-

139.

Money, A. G., Atwal, A., Boyce, E., Gaber, S., Windeatt,

S., & Alexandrou, K. (2019). Falls Sensei: A serious 3D

exploration game to enable the detection of extrinsic

home fall hazards for older adults. BMC Medical

Informatics and Decision Making, 19(1), 1-16.

Morán, A. I., Ramírez-Fernández, C., Meza-Kubo, V.,

Orihuela-Espina, F., García-Canseco, E., Grimaldo, A.

I., & Sucar, E. (2015). On the Effect of Previous

Technological Experience on the Usability of a Virtual

Rehabilitation Tool for the Physical Activation and

Cognitive Stimulation of Elders. Journal of medical

systems, 39(9), 1-11.

Orso, V., Spagnolli, A., Gamberini, L., Ibañez, F., &

Fabregat, M. E. (2017). Interactive multimedia content

for older adults: the case of SeniorChannel. Multimedia

Tools and Applications, 76(4), 5171-5189.

Palestra, G., Rebiai, M., Courtial, E., & Koutsouris, D.

(2019). Evaluation of a rehabilitation system for the

elderly in a day care center. Information (Switzerland),

10(1).

Pedroli, E., Greci, L., Colombo, D., Serino, S., Cipresso, P.,

Arlati, S., Mondellini, M., Boilini, L., Giussani, V.,

Goulene, K., Agostoni, M., Sacco, M., Stramba-

Badiale, M., Riva, G., & Gaggioli, A. (2018).

Characteristics, Usability, and Users Experience of a

System Combining Cognitive and Physical Therapy in

a Virtual Environment: Positive Bike. Sensors, 18(7).

Pripfl, J., Körtner, T., Batko-Klein, D., Hebesberger, D.,

Weninger, M., & Gisinger, C. (2016). Social service

robots to support independent living. Zeitschrift für

Gerontologie und Geriatrie, 49(4), 282-287.

Queirós, A., Dias, A., Silva, A. G., & Rocha, N. P. (2017).

Ambient Assisted Living and Health-Related

Outcomes—A Systematic Literature Review.

Informatics 2017, Vol. 4, Page 19, 4(3), 19-19.

Queirós, A., Silva, A., Alvarelhão, J., Rocha, N. P., &

Teixeira, A. (2015). Usability, accessibility and

ambient-assisted living: a systematic literature review.

Universal Access in the Information Society 2013 14:1,

14(1), 57-66.

Rebsamen, S., Knols, R. H., Pfister, P. B., & de Bruin, E.

D. (2019). Exergame-driven high-intensity interval

training in untrained community dwelling older adults:

A formative one group quasi-experimental feasibility

trial. Frontiers in Physiology, 10(AUG).

Martins, A. I., Rosa, A. F., Queirós, A., Silva, A., & Rocha,

N. P. (2015). European Portuguese validation of the

system usability scale (SUS). Procedia Computer

Science, 67, 293-300.

Sánchez-Morillo, D., Crespo, M., León, A., & Crespo Foix,

L. F. (2015). A novel multimodal tool for

telemonitoring patients with COPD. Informatics for

Health and Social Care, 40(1), 1-22.

Silva, A., Simões, P., Santos, R., Queirós, A., Rocha, N. P.,

& Rodrigues, M. (2019). A Scale to Assess the

Methodological Quality of Studies Assessing Usability

of Electronic Health Products and Services: Delphi

Study Followed by Validity and Reliability Testing.

Journal of medical Internet research, 21(11).

Teixeira, A., Ferreira, F., Almeida, N., Silva, S., Filipa

Rosa, A., Casimiro Pereira, J., & Vieira, D. (2017).

Design and development of Medication Assistant: older

adults centred design to go beyond simple medication

reminders. Universal Access in the Information Society,

16(3), 545-560.

Vaziri, D. D., Aal, K., Ogonowski, C., Rekowski, T. V.,

Kroll, M., Marston, H. R., Poveda, R., Gschwind, Y. J.,

Delbaere, K., Wieching, R., & Wulf, V. (2016).

Exploring user experience and technology acceptance

for a fall prevention system: results from a randomized

clinical trial and a living lab. European Review of Aging

and Physical Activity, 13(1), 1-9.

Werner, K., Oberzaucher, J., & Werner, F. (2012).

Evaluation of human robot interaction factors of a

socially assistive robot together with older people.

Proceedings - 2012 6th International Conference on

Complex, Intelligent, and Software Intensive Systems,

CISIS 2012, 455-460.

Wohlfahrt-Laymann, J., Hermens, H., Villalonga, C.,

Vollenbroek-Hutten, M., & Banos, O. (2019).

MobileCogniTracker: A mobile experience sampling

tool for tracking cognitive behaviour. Journal of

Ambient Intelligence and Humanized Computing,

10(6), 2143-2160.

Yilmaz, Ö. (2019). An ambient assisted living system for

dementia patients. Turkish Journal of Electrical

Engineering and Computer Sciences, 27(3), 2361-

2378.

Zhunio, C. S., Orellana, P. C., & Patino, A. V. (2020). A

Memory Game for Elderly People: Development and

Evaluation. 7th International Conference on

eDemocracy and eGovernment, ICEDEG 2020, 248-

252.

Zlatintsi, A., Dometios, A. C., Kardaris, N.,

Rodomagoulakis, I., Koutras, P., Papageorgiou, X.,

Maragos, P., Tzafestas, C. S., Vartholomeos, P., Hauer,

K., Werner, C., Annicchiarico, R., Lombardi, M. G.,

Adriano, F., Asfour, T., Sabatini, A. M., Laschi, C.,

Cianchetti, M., Güler, A., Kokkinos, I., Klein, B., &

López, R. (2020). I-Support: A robotic platform of an

assistive bathing robot for the elderly population.

Robotics and Autonomous Systems, 126.

A Scoping Review of the Inquiry Instruments Being Used to Evaluate the Usability of Ambient Assisted Living Solutions

327