Towards Elderly Social Integration using a Multimodal

Human-computer Interface

Vítor Duarte Teixeira

1,2

, Carlos Galinho Pires

, Fernando Miguel Pinto

João Freitas

1,2

, Miguel Sales Dias

1,2

and Eduarda Mendes Rodrigues

Microsoft Language Development Center, Porto Salvo, Portugal

ADETTI – ISCTE, IUL, Lisbon, Portugal

Dept. Informatics Engineering, University of Porto, Porto, Portugal

Abstract. This paper presents a multimodal prototype application that aims to

promote the social integration of the elderly. The application enables

communication with their social network through conferencing and social

media services, using natural interaction modalities, like speech, touch and

gestures. We begin by discussing the requirements and design guidelines that

were taken into account for the development of the prototype. We also present

the key elements of the development stage and the results of a usability study

conducted with ten elderly volunteers. The usability study reveals that such a

multimodal solution can simplify accessibility to the considered services.

Results indicate that this system is simpler, more natural and more enjoyable

than the current user interfaces. Furthermore, the natural interaction modalities

of the proposed prototype, allow elderly to be more efficient and have a better

user experience, thus contributing with an easier and faster way for this

population to join the information era.

1 Introduction

This paper proposes several ways for reducing the impact of social exclusion

observed in the elderly population. However, to effectively mitigate that issue, one

must take into account the fact that those individuals have developed resistance to

conventional forms of human-computer interaction, like the keyboard and mouse of

the WIMP paradigm, for instance, therefore making it necessary to test new natural

forms of interaction such as speech, touch and gestures. In addition, elder people

often have difficulties with motor skills due to health problems such as arthritis, so the

absence of small and difficult to handle equipment may be presented as an advantage

over current solutions. It is also known that due to ageing, senses like vision become

less accurate [15], hence difficulties in the perception of details or important

information in conventional graphical interfaces may arise, since current interfaces,

most notably in the mobility area, are not designed with these difficulties in mind.

There is also evidence that the European Union (EU) population is ageing rapidly.

The European Commission [5] estimates that by 2050 the elderly population in the

EU will be around 29% of the total population. This means that it is hastily becoming

Teixeira V., Pires C., Pinto F., Freitas J., Sales Dias M. and Mendes Rodrigues E..

Towards Elderly Social Integration using a Multimodal Human-computer Interface.

DOI: 10.5220/0003852800030013

In Proceedings of the 2nd International Living Usability Lab Workshop on AAL Latest Solutions, Trends and Applications (AAL-2012), pages 3-13

ISBN: 978-989-8425-93-5

 2012 SCITEPRESS (Science and Technology Publications, Lda.)

necessary to create solutions that allow overcoming the difficulties age brings to

people who want to use new technologies in order to remain socially active.

In this paper we present an adaptation made to the Living Home Center (LHC)

prototype developed by Pires et al. [13], which was specifically targeted at the

mobility impaired population. We have extended the services supported by that

prototype, by adding support for Instant Messaging, as well as integration with the

Facebook social network, enabling access to messages and photo albums. This paper

also presents two user studies, the first one that allowed gathering user requirements

for the development of the prototype and, the second one, which performed a usability

evaluation, aiming at testing if the proposed solution is actually useful to elderly

people.

The remainder of this document is structured as follows: Section 2 presents a

review of related work in the area of assistive technologies for seniors. Section 3

describes the results of the initial user requirements study, from which we derived

design guidelines and gathered knowledge about the elderly. Section 4 explains the

choices made during the prototype development and adaptation stage. Section 5

provides an analysis of the results obtained in the usability evaluation study and

presents a discussion of those results. Finally, section 6 presents the conclusions and

outlines future work.

2 Assistive Technologies for Seniors and Multimodal HCI

Elderly people who are connected to the world through the internet are less likely to

become depressed and have greater probability of becoming socially integrated [1].

However, despite being the population group that is more rapidly going online [12],

technological and interaction barriers still do not allow seniors to take full advantage

of the available services and content [18].

With the objective of allowing older people and people with disabilities to live

independently and be active in society, some initiatives have been launched by the

European Union. A good example is the e-inclusion project [4], which “aims at

reducing gaps in ICT usage and promoting the use of ICT to overcome exclusion, and

improve economic performance, employment opportunities, quality of life, social

participation and cohesion.”, by, for example, proposing a series of measures to

promote take-up of digital technologies by potentially disadvantaged groups, such as

elderly, less-literate and low-income persons.

Some studies, such as D'Andrea et al. [2] and Salces et al. [17], have demonstrated

that multimodal solutions can be used to improve the usage experience by a variety of

user groups, including the elderly. Therefore, if provided with the means of

multimodal interaction, elderly can experience improved accessibility to information

and an increased ability to socially integrate [16]. Multimodal User Interfaces

(MMUIs) also provide users with the ability to choose different modalities depending

on usage context, environment conditions or users special needs. This allows, for

example, improving stability and the robustness of speech recognition based systems.

The possibility to seamlessly alternate between input modalities is another advantage

of MMUIs, helping reducing the probability of injuries due to the overuse of a single

modality [10-11].

However, for the development of appropriate solutions with high accessibility one

must also take into account the specificities of universal design [10] as well as user

perspectives, thus avoiding the usage of inappropriate content, font or graphical

elements size, for example [7].

3 User Requirements Study

With the objective of gathering information about the main problems senior citizens

face while using the computer and current graphical user interfaces (GUIs) for

Communication (e.g., Windows Live Messenger (WLM)) and Social Media Services

(SMSs) (e.g., Facebook), we have conducted a requirements analysis study that

allowed the definition of a list of key requirements to be considered.

The study was divided in three parts. In the first part, we have performed a

structured interview using a questionnaire. In the second and third parts, we have

asked participants to carry out a set of tasks, first with existing communication and

social media services and afterwards with the LHC prototype, which was designed for

people with mobility impairments [13]. For the creation of the experimental tasks and

decision of the analysis methods we have followed the guidelines provided by Dix et

al. [3]. The goal of these tasks was to gather information about which modalities

would better satisfy the needs of the elderly. We have also aimed at gathering some

data concerning the participants’ reactions and opinions about the current LHC

version. By doing so, we wanted to find out which changes should be taken into

account for the newer version of the aforementioned system.

3.1 Study Participants

Ten elderly people took part in this requirements study. All the participants were

volunteers from the Lisbon University for Seniors (http://ul3i.com.sapo.pt/). The

selection of the participants was done randomly, however with a few restrictions, as

different computer skills and ages between participants and, especially, high literacy

levels were required.

The study group was composed of 2 males and 8 females, since it was difficult to

recruit an equal number of participants of each gender that fitted the desired profile.

The participants had an average age of 66.3 years old and different careers. As

requested, most of them presented high education levels, since that aspect “plays a

significant role in the way the user interacts with computers” [7].

3.2 Analysis of the Participants’ Skills and Habits, Via Questionnaire

The goal of this questionnaire was to determine computer, mobile phone and

smartphone frequency of use, nature of use and demonstrated skills, as well as habits

of usage of communication services and SMSs. This part was exclusively self-

assessing.

Interestingly, we have registered that the computer is not yet seen as a tool to

communicate with others, since participants mentioned having difficulties

understanding the current communication services UIs (e.g, WLM). They often said

that they could not get those kind of programs to work, or that they had a hard time

trying to add a contact to their friends list or establishing calls. The majority of the

considered sample also declared not using any SMS even though they knew of these

services existence. They argued that they heard too many times in the news and by

friends that using such a service might be dangerous and could lead to unpleasant

situations, if they did not know exactly what they were doing.

Concerning the usage of mobile devices, participants recognized using them,

however the adherence to smartphones was null, mainly due to confusing UIs, too

many features and small icons and text.

These results allow us to determine that our solution must provide access to an

easy-to-use communications service, with few and clear options, in order to mitigate

the difficulties mentioned above with current solutions. We also acknowledged that

the current SMSs have too many features, as far as elderly are concerned, and require

too much attention and skills. Therefore, we propose providing access only to the

features seniors considered most relevant (e.g., photo sharing and messages), and to

make the interface as easy and seamless to use as possible, thus trying to diminish the

observed feeling of insecurity while using such service.

We also intend to introduce a mobile application for smartphones that provides the

same services as the desktop version and attempts to overcome the difficulties

mentioned above.

3.3 Usability Evaluation of Current UIs for Communication and SMSs

On the second part of this first study we have asked the participants to do three

different tasks using current communication and social media services UIs: (1) instant

messaging task; (2) conferencing task and (3) Facebook task. The goal of these tasks

was to perform a usability evaluation of those kinds of interfaces concerning the

usage by senior citizens.

After the evaluation of an instant messaging task using WLM, we have confirmed

the results of the previous section. So, we have considered that it is important to

change the way information is shown. Text and buttons must be bigger, so it becomes

easier to find a contact or option. There should be buttons for tasks like opening a new

conversation and submitting an instant message, avoiding the need for double click,

double tap or usage of a special keyboard key, since the elderly have difficulties

performing and remembering those actions. Furthermore, it is important to have an

easy way of inserting special characters, since most participants were unable to insert

the ‘@’ and ‘€’ characters using the keyboard. Touch and speech should be

considered for this feature. Speech should work for command and control, but

especially for dictation when writing a message, since tests have demonstrated that

participants tend to be less productive when writing on a keyboard. It is also essential

that contacts are well identified, with name, a number and picture, if possible.

After completing a task that tested Facebook’s usability, participants confirmed that

this service had too many features, as mentioned in the previous section. We have

also confirmed that there is the need to make icons and text bigger and to organize it

better, since the current solutions UI elements are too small and do not always follow

the same logic of space organization, from screen to screen.

Tasks analysis revealed that participants took, in average, 1:29m (conferencing

task), 4:30m (instant messaging task) and 08:33m (Facebook task) more than the

control subject to complete the tasks. While differences between skilled and non-

skilled participants were 0:27m, 03:10m and 04:56m, , in the mentioned tasks. These

results allowed us understanding that some of the considered services do require some

level of experience, preventing less skilled users to be as effective while performing

the same tasks. We also registered that all the participants needed, in average, 2 aids

per task, so they could finish it.

3.4 Usability Evaluation of HCI Modalities using the Previous Version of LHC

On the third part of the first study, we have introduced participants to the previous

mobile and desktop versions of the LHC prototype, which were designed specifically

for mobility impaired individuals.

This part of the study allowed us to conclude that the elderly would be satisfied

with an application designed following similar guidelines as the previous LHC

version for mobility impaired individuals, since we have registered good levels of

performance and high levels of satisfaction when we have asked them to test it. We

have also observed that they normally felt much more relaxed using the prototype

than the previously tested solutions, so it became obvious that this application has the

potential to contribute for a more enjoyable user experience, concerning the elderly

individuals as well.

We have likewise determined that touch would be a critical modality to consider,

since the design of this application is completely adequate for touching, due to the

size and organization of the items. Nonetheless, we have considered speech as a must-

have modality too, as some of the elderly are less able to use touch, due to health

problems, such as arthritis [9].

4 LHC Prototype Adaptation

Based on the results of the user study we have gathered requirements that enabled us

to extend and improve the LHC prototype. As for specifications, we have focused

mainly on the improvement of the UI flow, on the support of more social media

services and on the prototype’s performance enhancement, while keeping the same

system architecture for the Living Home Center (LHC), with client, server and cloud

elements adhering to Microsoft technology, as described in the paper by Pires et al.

[13].

Even though no major changes on the desktop application UI were required, we

had previously learnt that users could benefit from the integration of Facebook’s

content in the prototype, since that is one of the most popular SMSs to date, with

currently more than 750 million active users, according to Facebook [6]. We have

realized that elderly would welcome also an instant messaging service, given its

popularity. Bearing that in mind, we have added features allowing access to

Facebook’s messages, profiles and also supporting media access and management,

such as photo albums. We have also added support for the insertion of comments and

likes on messages, albums or photos, from both the LHC users and from people in

their social network.

We have also developed a new version of the prototype for the Windows Phone 7

(WP7) platform (migrating from Windows Mobile 6.5), with increased stability of

that OS, the capacitive display of the newer devices, the larger memory and

processing power those same devices offer, as well as the ability to use multi-touch.

Figures 1 and 2, show the mobile and desktop versions GUIs of the enhanced LHC on

different contexts.

Fig. 1. Mobile Home Screen GUI – LHC.

Fig. 2. Desktop Photo Album Screen.

5 Usability Evaluation Study

We have conducted a usability study, following the method proposed by Preece et al.

[14], to evaluate how the enhanced LHC prototype performed, when used by a group

of elderly individuals. Additionally, we have gathered that group’s impressions about

the prototype and their evaluation about its value when considered as a tool to

promote social integration.

5.1 Study Participants

For this study we have asked for the collaboration of ten participants. This time those

were users of the Social Welfare Institute for the Armed Forces (http://www.iasfa.pt/)

in Lisbon, Portugal. For the selection of elderly, we have maintained the requirements

described on section 3.1.

The study group comprised 5 males and 5 females, with an average age of 77.9

years old and different careers. This time we have also requested for participants with

different literacy levels, since we wanted to register the differences between those

with lower and higher levels of education, while using the prototype. For calibrating

the study tasks and for comparing results, a neutral user, called the control subject,

performed the structured tasks. The control subject was a female with 22 years old,

who studied Marketing, had high computer skills and had never been presented to the

prototype before, hence being in the same conditions as the study participants.

5.2 Tasks and Methodology

Six different tasks were created to test different features of the improved LHC

prototype. Three tasks were designed to be performed on the mobile client and

another three designed for the desktop client. In both clients, tasks focused on the

usage of conference’s IM and video-call features, SMS profile feature, SMS messages

feature and SMS audio-visual feature. Besides hardware (keyboard and mouse),

different HCI modalities were always available for each task and client platforms,

such as speech, touch and 3D gestures in certain mobile application contexts.

To be able to gather relevant information and maintain the same conditions for

every participant, a protocol was created and followed for all the sessions. The

protocol creation was guided by the work of Dix et al. [3].

After completing all tasks, each participant answered a final questionnaire, with

open and closed questions, with the objective of determining the satisfaction with

each HCI modality in terms of easiness and enjoyment, and also to evaluate the

efficiency of the UI and the considered features.

In order to evaluate a task, both qualitative and quantitative results were taken into

consideration, following the same recommendations as the tests made with the

previous version of this prototype [13]. Although quantitative results were gathered

these must not be seen as the most relevant data, because of the reduced number of

participants considered, the sample is statistically unrepresentative. However, it

should be seen as a pre-test or a guide, to a future test with a more representative

sample.

5.3 Tasks Results

Results for the mobile application tasks show that the difference on the average times

of execution, between the control subject and the participants varied from 1:14m to

1:56m, while differences between skilled and non-skilled users would not exceed 35

seconds. The task that required the visualization of the received and sent social

messages was the one with the better results, as depicted in Figure 3.

The number of aids was similar to all the tasks. Only 5 participants out of 10

requested help, and the aids did not exceed 1 for each one of those participants. The

preferred modality of interaction was Touch, with an average of 10 uses per task per

participant. Most participants argued that the “push-to-talk” (PTT) mechanism

available in the mobile version was complicate and was of little help for achieving the

desired action. Other issues pointed to the mobile application included the fact that

the virtual keyboard is not large enough to enable elderly users to hit a key at a time

and also, the fact that some buttons are hidden and not obvious enough, such as the

Save Profile button in the SMS’s Profile screen.

Desktop tasks analysis revealed that participants took, in average, from 1 minute to

4:26m more than the control subject to complete the tasks, whereas computer skilled

participants were 12 to 36 seconds faster than non-skilled participants.

The task that required the users to establish a video-call with a specific contact was

the most enjoyed and that feature was reported as the most useful in the application. It

was also the fastest task and the one with fewer differences between user categories,

as shown in Figure 4. The video-call task was also the one that registered less aids,

with only 5 participants requesting help one time. The remaining tasks registered the

average of an aid per participant per task. However, if we consider that those aids had

mostly to do with them asking how to bring up the built-in virtual keyboard, we could

overlook these data.

Fig. 3. Mobile social messages task average

execution times.

Fig. 4. Desktop video-call task average

execution times.

The preferred modality of interaction was touch, followed by speech. In average, it

was used 3 times more than the speech modality and 15 times more than hardware

(keyboard and mouse). The only issue reported by users had to do with the fact that

dictation with the current European Portuguese speech recognition engine is still far

from being 100% accurate, especially with the elderly voice, since a parallel project

(LUL) is still on-going with the objective of collecting those voice characteristics for

the augmentation of the aforementioned engine capabilities. Satisfaction with the

appearance and organization of the UI was also noticeable, as most participants

commented that the appearance and organization of the screens was appealing and

easy to understand.

5.4 Results Discussion – Requirements Study vs. Usability Study

At this point we could compare times of execution for similar tasks when using

currently available UIs for communication and SMS services, in one hand and LHC,

in the other. Concerning that case, we have considered only tasks performed with

desktop clients, since in the requirements study, we did not consider the mobile

versions of those UIs.

As shown in Figure 5, in general LHC has allowed users, to be more effective

performing all tasks, when compared to the existing GUIs of Windows Live

Messenger (IM and Conference) and Facebook (SMS).

Fig. 5. Requirements study results vs. usability study results.

Not only did users take less time performing tasks, they also needed less aids when

using LHC. As referred above they needed, in average, 1 aid per task using LHC,

opposed to the 2 aids per task using other UIs.

5.5 Questionnaire Results

The questionnaire was based on the one applied on the previous usability study made

with mobility impairment users [13]. Participants generally considered all modalities

easy or reasonably easy to use, and only on some cases difficult. Participants enjoyed

using almost all modalities, tending to prefer touch and speech on the desktop version.

Eight participants manifested preference for touch on the desktop application. Speech

on the desktop application, was selected by 4 participants. The less enjoyed

modalities were touch and 3D gestures on the smartphone. We have asked

participants if they felt the prototype interface was easy to use, obtaining positive

response from all the participants. The preferred version of the prototype was,

unanimously, the one developed for desktop computers.

When asked what they thought we should change or add to the prototype, so it

would behave according to their needs, participants simply responded that they’d like

us to change the mobile keyboard layout and enhance (European Portuguese) speech

recognition for command and control and, especially, for dictation.

6 Conclusions and Future Work

The usability evaluation allowed us to derive the following conclusions:

(1) The prior prototype GUI was actually adequate for the elderly too, being

classified as simpler, more natural and more enjoyable than current UIs (e.g., WLM

and Facebook). (2) Multimodality and universal design play an important part on

fighting e-exclusion, confirmed by the fact that all the participants revealed to be

relaxed, entertained and effectively using the prototype, achieving relatively

complicated actions with little effort. Also, allowing different users to choose the

modality that was most suited on different contexts, made the application being usable

for both low and high skilled participants. (3) The availability of touch, on both

clients, is very important and has a direct impact on the elderly performance and

level of satisfaction. (4) Speech is seen as an important modality, especially for

individuals that have issues using their arms or hands. However, because the accuracy

of the current speech engine for European Portuguese for elderly is still undergoing

improvements, participants became somewhat disappointed with the current dictation

capability, describing its enhancement as a must-have feature and a good solution to

their limitations when trying to use a keyboard. (5) The desktop client version is the

one that better suits the current elderly population, since its use did not present

difficulties to most of the participants. Regarding the mobile version, there are some

improvements to be made, such as the simplification and augmentation of the virtual

keyboard. The latter was also seen as a tool for the elderly of the future, since those

will have more experience with such devices. (6) The prototype services are useful

in fighting social exclusion, improving the elderly ability to communicate and

share. As an example, users regarded the usage of the video-call feature on LHC as

much simpler than with common systems, and believed it would be one the most

useful services to enable communication with family and friends.

Bearing in mind that the control subject had much higher computer skills than all

the participants, the results obtained in the usability study can already be interpreted

as very good indicators for further research. The comparison between current UIs for

communication and SMSs and the LHC prototype also gave good pointers for the

evaluation of the overall LHC performance.

Future work should consider the improvement of speech recognition for the

elderly, focusing especially in the dictation mode, improving the desktop UI design

and the mobile version’s virtual keyboard. It’s also desirable to investigate if the

prototype would be welcomed and usable in a real live scenario and if these results

are repeatable with a larger group of users.

Acknowledgements

This project was co-funded by Microsoft, under QREN Living Usability lab (http://

www.livinglab.pt/), the national program of incentives for the Portuguese businesses

and industry, QREN (in the scope of its R&D incentive program), in the context of

the Operational Program for Competitiveness Factors. The authors would like to

thank the Lisbon University for Seniors (http://ul3i.com.sapo.pt/) and to the Social

Welfare Institute for the Armed Forces (http://www.iasfa.pt/), for providing the means

to recruit the users for the user requirements and usability evaluations studies.

References

1. Cisek, E. and Triche, K. (2005). Depression and Social Support Among Older Adult

Computer Users, August 18, 2005, 113th Annual Convention of the American

Psychological Assoc.

2. D'Andrea, A., D'Ulizia A., Ferri, F. and Grifoni, P., (2009). A multimodal pervasive

framework for ambient assisted living. New York, NY, USA, PETRA '09: Proceedings of

the 2nd International Conference on PErvsive Technologies Related to Assistive

Environments.

3. Dix, A., Finlay, J., Abowd, G.D. and Beale, R. (2003). Human-Computer Interaction (3rd

ed.). Essex: Prentice Hall

4. Europe’s Information Society (2010). e-Inclusion Visited July 14, 2011, at

http://ec.europa.eu/information_society/activities/einclusion/index_en.htm

5. European Commission (2010), Europe in figures – Eurostat yearbook 2010, Retrieved July

10, 2011, from http://epp.eurostat.ec.europa.eu/portal/page/portal/product_details/

publication?p_product_code=KS-CD-10-220

6. Facebook (2011), Press Room – statistics, Visited July 15, 2011, at http://

www.facebook.com/ press/info.php?statistics

7. Júdice, A., Freitas, J., Braga, D., Calado, A., Dias, M.S., Teixeira, A., Oliveira, C. (2010).

Elderly Speech Collection for Speech Recognition Based on Crowd Sourcing. In

Gnanayutham, P., Paredes, H., Rekanos, I. T. (Eds), Proceedings of DSAI2010

International Conference (pp. 103-110).

8. LUL, Lab Living Usability Lab for Next Generation Networks, www.livinglab.pt, visited

July 15, 2011

9. Makover, M.E., Zieve, D., (2011), Arthritis - Joint inflammation, visited July 15, 2011, at

http://www.ncbi.nlm.nih.gov/pubmedhealth/PMH0002223/

10. Oviatt, S. (2001). Designing robust multimodal systems for universal access, Proceedings

of the 2001 EC/NSF workshop on Universal accessibility of ubiquitous computing:

providing for the elderly event report. p. 71–74.

11. Oviatt S., Cohen, P., Wu, L., Vergo, J., Duncan, L., Suhm B., Bers, B., Holzman, T.,

Winograd, T., Landay. J., Larson, J., and Ferro, D. (2000). Designing the user interface for

multimodal speech and pen-based gesture applications: state-of-the-art systems and future

research directions. Hum.-Comput. Interact. 15, 4 (December 2000)

12. Pew Internet & American Life (2006), Are Wired Seniors Sitting Ducks?, Retrieved July

10, 2011, from: http://www.pewinternet.org/Reports/2006/Are-Wired-Seniors-Sitting-

Ducks.aspx

13. Pires, C., Pinto, F., Rodrigues, E. M. and Dias, M. S. (2011). On the benefits of speech and

touch interaction with communication services for mobility impaired users. 28-29 January,

2011, Rome, Italy, 1st International Living Usability Lab Workshop on AAL Latest

Solutions, Trends and Applications - AAL 2011

14. Preece, J., Rogers, Y. and Sharp, H. (2002). Interaction Design: Beyond Human-Computer

Interaction. Wiley, First edition.

15. Quillen, D.A., (1999). Common Causes of Vision Loss in Elderly Patients, Retrieved July

10, 2011, from: http://www.aafp.org/afp/990700ap/99.html

16. Richter, K. and Hellenschmidt, M. (2004) Interacting with the ambience: Multimodal

interaction and ambient intelligence. France, W3C Workshop on Multi-modal Interaction

17. Salces, F.J.S.d., D. England, and D. Llewellyn-Jones (2005). Designing for all in the house.

CLIHC ’05: 2005 Latin American conference on HCI. 2005. p. 283–288.

18. Sfyrakis, S.A., Stephanidis, C., Akoumianakis D., Sfyrakis M., Paramythis A. and Crete

H. (1998). Universal accessibility in HCI: Process-oriented design guidelines and tool

requirements. 4th ERCIM Workshop on "User Interfaces for All". p. 19-21.