AUGMENTED OBJECTS TO SUPPORT PEOPLE WITH MILD

COGNITIVE DEFICIENCIES IN EVERYDAY ACTIVITIES

Andrea Piras, Davide Carboni, Antonio Pintus

CRS4, Center for Advanced Studies, Research and Development in Sardinia, Ed.1 Polaris, 09010 Pula (CA), Italy

Sylvain Giroux

Laboratoire DOMUS, University of Sherbrooke, 2500, Boulevard de l’Université, J1K 2R1 Sherbrooke, QC, Canada

Keywords: Cognitive deficiencies, Tangible user interfaces, Augmented object, Visual tag, Activity of daily living,

Kitchen.

Abstract: The cognitive deficiencies are the spectrum of deficiencies related to the reduction of brain cognitive

capability. They mainly affect decision-making, reasoning, planning and solving capabilities of daily

activities. The consequence is cognitively impaired people become dependent by families, caregivers or

Health Institutes. Not-intrusive information technology applications can be applied onto augment the house

of mild cognitive impaired people to maintain their autonomy and facilitate their daily activities. At the

same time, the applications combat the deficiency effects and are a good mean to promote self-esteem and

increase the independent time living at home. In this paper, we propose an approach to use everyday objects

as input and output devices for a processing system supporting people affected by mild cognitive

deficiencies in successfully complete activities of daily living.

1 INTRODUCTION

Cognitive is an adjective related to the abstract

human faculty of information process and

knowledge applying closely related to concepts as

reasoning, comprehension, inference, decision-

making, planning and others capabilities of the

human mind.

Example of such diseases are traumatic brain

injury, dementia, Alzheimer's disease, mental

retardation, schizophrenia and autism. That

fundamental abilities are necessaries to successfully

complete the common daily activities so related

deficiencies have the effect to compromise any

action with the consequence people becomes

dependent by families, caregivers and National

Health Institutes.

This dependence raises a set of problems for all

involved actors. Families should face with the time

required to assist the impaired relative and the cost

for personal caregivers.

The National Health Institutes support the

families and patient providing appliances, ad-hoc

public structures and specialized medical staffs.

Elements drawing on national economies and state

budgets. Last and more important, it is the patients

acceptance of her disease and of the helps from

relatives and institutes. They often are angry, do not

accept the disease, refuse to use medical appliances

installed in their house, consider an intrusion the

continuous presence of caregivers and hate the lost

of independence.

A solution able to satisfy all of them consists on

augmenting the patient's house with not-intrusive

Information Technology (IT) applications. Where

the whole house becomes the computing

environment actively supporting people in daily

living activities either serving their needs, providing

their safety and monitoring their actions.

Each object, household appliance, furniture and

any other element becomes an input source,

contributing to compose the user interface for

interacting with the digital processing system. The

outputs are indications or helps exposed on or by the

same elements providing input.

In this paper, we propose an approach to use

everyday objects as input and output devices for a

processing system supporting people affected by

213

Piras A., Carboni D., Pintus A. and Giroux S..

AUGMENTED OBJECTS TO SUPPORT PEOPLE WITH MILD COGNITIVE DEFICIENCES IN EVERYDAY ACTIVITIES.

DOI: 10.5220/0003149702130218

In Proceedings of the International Conference on Health Informatics (HEALTHINF-2011), pages 213-218

ISBN: 978-989-8425-34-8

 2011 SCITEPRESS (Science and Technology Publications, Lda.)

mild cognitive deficiencies in successfully complete

Activities of Daily Living (ADLs). Next sections

will introduce to the tangible user interfaces defining

such a type of object-based interaction and the IT

architecture of the processing system. The attention

is focused on the daily activities performed in the

kitchen and the ingredients/utensils collection is the

activity analyzed as example.

2 TANGIBLE USER INTERFACES

To support people with mild cognitive deficiencies

in everyday activities, traditional input-output

interaction system based on graphical user interfaces

(GUIs) looks to be not adequate because rely on

input devices keyboard, mouse or touch-screen and

do not explore the natural ability to interact

manipulating physical objects. Such a gap is filled

by the tangible user interfaces (TUIs).

They “provide physical form to digital

information and computation, facilitating the direct

manipulation of bits” (Ishii, 2008). Physical,

touchable, concrete objects are used as

representations of digital information and

computational operations. User executes physical,

real and direct manipulations of these objects to

interact with processing systems.

In everyday live, humans interact with the

physical world processing an associating of

symbolic functions and relationships with physical

artifacts. Boards games like draughts, mahjong and

Nine Men Morris are some examples where

distinctive physical tokens represent people,

physical entities or actions.

The abacus, the oldest human developed tool

(some hypothesis date it back to 8000 BC, before

written language and even the wheel) (Schmandt-

Besserat, 1997), is a TUI where tokens changes their

values according the position on the board.

2.1 Augmenting Everyday Objects

Objects are fundamental parts of the everyday life of

everybody independently by his religion, country,

gender, age or ability. We actively surround

ourselves with objects, helping to establish our

identities and to find arrangement of those objects

such as to constitute an autotopography (a physical

map of memory, history and belief) (Gonzalez,

1995).

TUIs put the everyday objects to the centre of the

attention to interact with digital systems reducing or

cancelling the entering barrier of the manual

dexterity of the digital input devices. The connection

between the physical objects and the digital

processing is fundamental.

It is realized coupling the objects with a reader.

The reader is able to recognize the object, read its

information and forward it to processing system.

Objects could be augmented inserting

microelectronic circuits or Auto-ID technologies

(e.g. RFID or visual tag). We prefer to pursuit the

“augmentation” in the simplest and no-intrusive

way, so visual tags are the ideal choice because it is

sufficient to paste a little draw on the object surface.

In particular, Quick Response (QR) (Denso,

2010) and Amoeba, from ReacTIVision framework

(Kaltenbrunner and Bencina, 2007), are kinds of

visual tags that can fast recognized by video streams

using an accompanying detection algorithm.

Figure 1: Example of QR (on the left) and Amoeba visual

tags.

3 ARCHITECTURE

According to the pipes run-time architecture (Pintus

et al, 2010), to design the architecture we make the

assumption that each augmented object is coupled to

a Web service. It describes the object features, what

actions it can perform (if able to be an effector or

actuator) and what data it provides according user

manipulation (if able to be a sensor). The set of

available augmented objects defines the sensor

network.

The visual tag placed on the object contains a

reference to the Web address of the related Web

Services Description Language - WSDL (WSDL,

2007) instance. To add one augmented object to the

sensor network, we adopt the point-click interaction.

When technicians sets the system, he points the

visual tag with a mobile device and by clicking the

visual tag is decoded and the sensor network is

commanded to acquires the related Web service.

The task manager uses the set of Web services

registered in the sensor network to compose the task

related to the ADL, including the actions to perform

in case of wrong or missing action by the patient. It

can modify, deploy and remove tasks.

HEALTHINF 2011 - International Conference on Health Informatics

214

Figure 2: Schema of the processing system using the augmented objects.

The deployed tasks are performed by the task

processor. It is a WS-BPEL engine able to compose,

dynamically deploy and execute the WS-BPEL

process (Oasis, 2007) based on the defined task and

the WSDL URLs of the involved objects. The tasks

idle waiting for input by augmented objects and

forwarded by the sensor network.

According to such input, the task processor

performs the next step activating selected effectors

and actuators by the invocation of methods on their

Web services.

4 A CASE STUDY:

THE AUGMENTED KITCHEN

The kitchen is the room of the house where people

spend a lot of time performing actions and

interacting with others and it is ideal to test TUI

solutions on account of the tools and utensils used

for cooking. The successful completion of meal

preparation is not only a need for a healthy eating

but also an important step toward the independence

and increase self-esteem for mild cognitive impaired

people.

To augment the kitchen, we need to register the

sensors and effectors composing the sensor network.

Firsts are the input elements of the task processor

and any everyday object with a visual tag becomes a

sensor able to provide input. Effectors are electronic

objects able to show or play an output.

After sensor network setting, we can count on it

and a smart task processor, but we must remember

to let people to interact in the most simple possible

way defining the TUI with high attention in order to

be as intuitive as possible and accepted by people.

LEDs and lights are our privileged outputs, leaving

on traditional screen eventually alerts on wrong or

missing user actions and the video indications to

address them.

4.1 An Example of Task:

The Meal Preparation

Picture cards and card games are related to fun time

of the life of any one since we were children. With

the Virtual Kitchen cards – VKards (Piras et al,

2009), we want to reuse that experiences and the

pleasure to play with such games in order to help

and train the mild cognitive impaired people on

recipe selection and ingredient/utensil search. The

user interaction with VKards becomes input for the

task processor.

We define two kinds of VKards: recipe and

ingredient/utensil. Each card is structured according

to the visual supports defined by Picture Exchange

Communication System (PECS) (Bondy and Frost,

2001). Such a system is a unique

augmentative/alternative training package that

teaches children and adults with autism and other

communication deficits to initiate communication

using simple pictures exposing one image and its

related text.

Each VKard depicts one image, the name of its

subject, plus eventually the quantity, and one

Amoeba visual tag on the top right corner. The

AUGMENTED OBJECTS TO SUPPORT PEOPLE WITH MILD COGNITIVE DEFICIENCES IN EVERYDAY

ACTIVITIES

215

recipe VKard is recognizable by the empty dish and

the text “RECIPE” on its top. The dish background

has the same colour of the borders of

ingredients/utensils VKards used in that recipe. The

same colour fills the cards back.

Figure 3: Example of VKards. From left to right: the

recipe, one ingredient and one utensil.

The interaction model is the Token+Constraint

(Ullmer at al, 2005). The VKard is the token and the

constraints are areas where place the cards: the

recipe area where placing the recipe card, the search

area for the object that person is searching in the

kitchen, the missing items area for items not

available and the found items area.

5 RELATED WORKS

Around the world, several smart homes has been

realized. Casattenta (T3Lab, 2010) and House

n/PlaceLab (House_n, 2010) investigate how

fabrication and sensing tools can be used to create

responsive, adaptable environments helping in

everyday life, integrating safety, comfort and energy

saving where everything is under control but no

more restrictions in everyday actions.

eKoti (DE, 2010) are smart home projects to

design and construct electronic devices that user

could not perceive, until he would actually use them.

Gator Tech Smart House (Gator Tech, 2010) is a

laboratory-house designed to assist older and

disabled people in maximizing independence and

maintaining a high quality of life (Helal et al., 2008).

In Canada, the DOMUS laboratory (DOMUS, 2010)

of the University of Sherbrooke studies and develop

solutions for cognitive assistance conjugating an

intelligent home with mobile applications. The

Ambient Kitchen (Olivier et al, 2009), placed in

Newcastle, is a laboratory based on a real kitchen

with a wide set of sensors and digital equipment in

order to evaluate pervasive computing prototypes

able to support people with cognitive impairments

Other works focused their attention on meal

preparation. Counteractive (Ju et al., 2001) teaches

cooking through a fixed multimedia projection of

recipe steps and contextual information is neither

used nor available. The Virtual Recipe of the

Chameleon Kitchen (Lee, 2005) uses numerous

invasive projectors and camcorders raising several

occlusion problems to expose the predefined

sequence of steps.

Other kinds of technologies can be used to

enhance objects:

 Bluetooth emitters (Salminen, Hosio, and Riekki,

2006);

 Infra-Red tags (Ailisto et al., 2006);

 barcodes, like AURA (Bernheim Brush et al.,

2005) and WebStickers (Holmquist et al., 1999);

 and RFID.

The last one is the major competitor of visual tags.

Even if the RFID theory has been published in the

60’s, only since 90’s it started to be widely used on

packages, blood sacks (Sandler et al., 2006), and so

on. Passive RFID tags and visual tags are both so

cheap. RFID requires specific readers while for

visual tags a camcorder is sufficient. Both of them

have occlusion problems but in favor of visual tags

there is the fact they do not get interferences by

metals and liquids.

6 CONCLUSIONS

The number of people affected by cognitive

deficiencies is predicted to increase a cause of the

population aging. The World Health Organization

(WHO) provided an estimation of approximately

24.5 millions of people affected with dementia in

2005, which was going to increase to 30 million in

2015, reach 45 millions by 2030 (WHO, 2006) and

81 million by 2040 (ADI, 2008). Another study

provided a worldwide estimation of AD in 2006 of

26.6 million which would multiply by four in 2050

(Brookmeyer et al., 2007).

Regarding economic problems, family caregivers

often give up time from work to spend 47 hours per

week on average with the person with AD. Old

assessments, (Ernst and Hay, 1994), (Meek et al.,

1998) evaluated the direct and indirect annual costs

of caring for people with dementia is at least $100

billion per year in the United States.

Such numbers justify the great effort performed

by Governments, industries and researchers to find

solution to reduce the impact of cognitive

deficiencies allowing individuals to best live in their

HEALTHINF 2011 - International Conference on Health Informatics

216

homes, mixing and merging interdisciplinary

technologies and studies, as architecture, design,

electricity, ergonomics, electronics, computer

science and psychology.

On our opinion, augmenting objects with not-

intrusive technologies and defining TUI that relies

on common activities are fundamental elements to

design tasks able to enhance autonomy, to combat

the deficiency effects and a good mean to promote

self-esteem.

With respect to the interaction with traditional

GUI, the TUI lets people to take advantage of the

human haptic interaction skills for manipulating

physical objects. For people aged or with mild

cognitive deficiencies the use of common input

devices may represent another difficult tasks, and

TUI represents a more suitable alternative.

Furthermore, it has the not secondary effects to

stimulate the manual interaction with everyday

objects and to push them to perform action and

instead to stay passive.

Specifically on the use of VKard, the advantages

are:

 its extreme cheapness (it is sufficient to print

them);

 the ingredients and utensils can be searched any

time and with the user preferred order;

 the cognitive deficient person focuses attention

on the card, its picture and its text without striving to

associate recipe and ingredients, names with mental

images or pictures.

Some drawbacks are:

 the need to one constraint area;

 occlusions can not let the camcorder to see the

visual tag and so to detect the VKard.

The proposed approach is an on-going work that will

require several experimentations considering the

particular the end-user category and the number of

different disciplines involved. Before to formulate it,

we have studied what could let to realize it and we

lend TUI and visual tags from IT world, the PECS

visual supports from alternative communication and

VKards from card games. After the development

phase, we plan to perform test sessions with normal

and deficient cognitive people.

REFERENCES

Ailisto, H., Pohjanheimo, L., Välkkynen, P., Strömmer,

E., Tuomisto, T., and Korhonen, I. (2006). Bridging

the physical and virtual worlds by local connectivity-

based physical selection. Personal and Ubiquitous

Computing, 10(6), 333-344.

ADI - Alzheimer's Disease International (2008). The

Global Impact of Dementia. Retrieved July 6, 2010,

from http://www.alz.co.uk/media/dementia.html.

Bernheim Brush, A. J., Combs Turner, T., Smith, M. A.

and Gupta, N. (2005). Scanning objects in the wild:

Assessing an object triggered information system. In

Ubi-comp 2005, 305-322. Springer-Verlag.

Bondy, A. S., and Frost, L. (2001). The Picture Exchange

Communication System. Behavior Modification, 25

(5), 725-744.

Brookmeyer, R., Johnson, E., Ziegler-Graham, K., and

Arrighi, M. H. (2007). Forecasting the global burden

of Alzheimer’s disease. Alzheimer's and Dementia,

3(3), 186-191.

Denso (n.d.). About QR. Retrieved July 6, 2010, from

http://www.denso-wave.com/qrcode/aboutqr-e.html.

DE - Department of Electronics – Tampere University of

Technology (n.d.). eKoti. Retrieved July 6, 2010, from

http://www.ele.tut.fi/research/personalelectronics/proj

ects/ekoti_03/index.htm.

DOMUS laboratory - Research in Domotics and Mobile

Computer Science, University of Sherbrooke.

Retrieved July 6, 2010, from

http://domus.usherbrooke.ca/?locale=en.

Ernst, R. L., and Hay, J. W. (1994). The U.S. Economic

and Social Costs of Alzheimer's Disease Revisited.

American Journal of Public Health, 84(8), 1261 -

1264.

Gator Tech, MPCL - Mobile and Pervasive Computing

Laboratory, University of Florida (2010). Gator Tech

Smart House. Retrieved July 6, 2010, from

http://www.icta.ufl.edu/gt.htm.

Gonzalez, J. (1995). Autotopographies. Prosthetic

Territories – Politics and Hypertechnologies, 133-150.

Helal, A., King, J., Zabadani, H., and Kaddourah, Y.

(2008). The Gator Tech Smart House: An Assistive

Environment for Successful Aging. Advanced

Intelligent Environments.

Holmquist, L. E., Redström, J. and Ljungstrand, P. (1999).

Token-based access to digital information. In 1st

International Symposium on handheld and ubiquitous

computing. 234-245. Springer-Verlag.

House_n Research Group, Department of Architecture,

Massachusetts Institute of Technology (n.d.). House n.

Retrieved July 6, 2010, from http://architecture.mi

t.edu/house_n.

Ishii, H. (2008). Tangible bits: beyond pixels. In 2nd

International Conference on Tangible and Embedded

Interaction.

Ju, W. et. Al. (2001). Counteractive: An Interactive

Cookbook for the Kitchen Counter. In CHI 2001, 269-

270.

Kaltenbrunner, M., and Bencina, R. (2007). reacTIVision:

A Computer-Vision Framework for Table-Based

Tangible Interaction. In

1st International Conference

on Tangible and Embedded Interaction (TEI07)

AUGMENTED OBJECTS TO SUPPORT PEOPLE WITH MILD COGNITIVE DEFICIENCES IN EVERYDAY

ACTIVITIES

217

Lee, C. H. (2005). Spatial User Interfaces: Augmenting

Human Sensibilities in a Domestic Kitchen.

Massachusetts Institute of Technology

Meek, P. D., McKeithan, K., and Schumock, G. T. (1998).

Economic considerations in Alzheimer's disease.

Pharmacotherapy, 2(18)

Olivier, P., Xu, G., Monk, A., and Hoey, J. (2009)

Ambient kitchen: designing situated services using a

high fidelity prototyping environment. In Proceedings

of the 2nd international Conference on Pervasive

Technologies Related To Assistive Environment -.

PETRA '09, 1-7.

OASIS Web Services Business Process Execution

Language Version 2.0 (2007). Retrieved July 6, 2010,

from http://docs.oasis-open.org/wsbpel/

2.0/wsbpel-v2.0.pdf.

Pintus, A., Carboni, D., Piras, A., and Giordano, A.

(2010). Connecting Smart Things through Web

services Orchestrations. In Post-Proc. Of ICWE 2010,

TouchTheWeb’10 Workshop. Springer.

Piras, A., Giroux, S., and Bauchet, J. (2009). Playing

VKards to support cognitively impaired people on

meal preparation. In Adjunct Proc. of AmI09. 277-280.

ISBN 978-3-902737-00-7.

Salminen, T., Hosio, S., and Riekki, J. (2006). Enhancing

Bluetooth Connectivity with RFID. In 4th Annual

IEEE International Conference on Pervasive

Computing and Communications, 36-41. IEEE

Computer Society.

Sandler, G., Langeberg A., Carty K., Laurie J., and

Dohnalek, L. J. (2006). Bar Code and Radio-

Frequency Technologies Can Increase Safety and

Efficiency of Blood Transfusions. Laboratory

Medicine, 37(7), 436-439.

Schmandt-Besserat (1997). How Writing Came About.

Austin: University of Texas Press.

T3Lab (n.d.), Casattenta. Retrieved July 6, 2010, from

http://www.t3lab.it/casattenta/page22.htm.

Ullmer, B., Ishii, H., and Jacob, R. J., (2005).

Token+constraint systems for tangible interaction with

digital information. ACM Transaction Computer-

Human Interaction, 12(1), 81-118.

WSDL Version 2.0 Part 0: Primer, (2007). Retrieved July

6, 2010, from http://www.w3.org/TR/2007/

REC-wsdl20-primer-20070626/.

WHO - World Health Organization (2006). Neurological

Disorders: Public Health Challenges. 204-207.

Retrieved July 6, 2010, from http://www.who.int/

mental_health/neurology/chapter_2_neuro_disorders_

public_h_challenges.pdf.

HEALTHINF 2011 - International Conference on Health Informatics

218