A Music Tangible User Interface for the Cognitive and Motor
Rehabilitation of Elderly People
Adriano Barat
, Helene Korsten and Luca A. Ludovico
Laboratory of Music Informatics, Department of Computer Science, University of Milan, via G. Celoria 18, Milan, Italy
Music, Tangible User Interface, Web, Kibo.
This paper proposes the combined use of tangible user interfaces, digital technologies, and musical expression
in the context of cognitive and motor rehabilitation of elderly people. After analyzing the state of the art about
common age impairments and tangible user interfaces in rehabilitation, we will introduce the Kibo, a MIDI
controller based on the concept of fiducials and able to communicate with other MIDI devices via Bluetooth.
The peculiar characteristics of the Kibo will be exploited in a Web framework aiming to foster the development
or recovery of cognitive and motor abilities through 3 specially designed games. The paper will also report
remarks from domain experts (music therapists and physiatrists) and the consequent redesign.
According to a 2019 report by the United Nations,
there were 703 million persons aged 65 years or over
in the world in 2019 and the number of older persons
is projected to double to 1.5 billion in 2050. These
changes for individuals are mirrored in society since
older persons represent a growing demographic group
in society. One of the goals for sustainable devel-
opment should be investing in education, health, and
well-being for all, including the elder.
The target audience of the project documented
in this paper is represented by elderly people who
need rehabilitation sessions or are dissatisfied or in-
active in their social interactions. Starting from
an analysis of the most common impairments and
changes in cognitive and motor abilities due to age,
we designed and developed a prototype applicable to
music-therapy rehabilitation sessions. Interventions
can address a variety of healthcare and educational
goals (e.g., promoting wellness, managing stress, al-
leviating pain, expressing feelings, enhancing mem-
ory, improving communication, promoting physical
rehabilitation, etc.).
The result is a Web-based prototype that offers in-
tuitive interaction with musical parameters via a spe-
cific tangible user interface, called the Kibo, so as to
take advantage of the haptic interaction skills with the
environment. In particular, three browser games have
been developed in order to train rhythm, spatial, and
recognition skills.
In this section, we will review the scientific literature
that is most relevant for our purposes. Due to the vast-
ness of the subject, we cannot be exhaustive. We will
limit ourselves to a few references of particular im-
portance for our work, addressing: i) common age
impairments, both cognitive and physical; ii) tangi-
ble user interfaces and their applicability to the music
field; and iii) experiences of tangible user interfaces
in rehabilitation.
Starting from age impairments, due to an in-
creasing life expectancy, the number of elderly peo-
ple has increased significantly worldwide. Thus, so-
ciety will have to offer more and more products and
services that meet the specific needs and desires of
a geriatric age group that makes up a great percent-
age of the population. These people are seeking solu-
tions to help them cope with daily life, give them the
opportunity to interact socially, and find alternative
ways of entertainment and learning. Moreover, health
systems are dealing with the ever-increasing burden
of finding solutions and cures for age-related and de-
generative conditions (e.g., dementia and Alzheimer’s
disease) and impaired-movement pathologies (e.g.,
Baratè, A., Korsten, H. and Ludovico, L.
A Music Tangible User Interface for the Cognitive and Motor Rehabilitation of Elderly People.
DOI: 10.5220/0011395900003323
In Proceedings of the 6th International Conference on Computer-Human Interaction Research and Applications (CHIRA 2022), pages 121-128
ISBN: 978-989-758-609-5; ISSN: 2184-3244
2022 by SCITEPRESS Science and Technology Publications, Lda. All rights reserved
Everyone experiences changes as an inevitable
part of the natural degeneration associated with ag-
ing (Iversen, 2015). Concerning changes in cognitive
abilities, aging causes a decline in spatial cognition,
which is the ability to represent spatial relationships
among objects. The results of a study conducted in
2008 by Iachini et al. show that some spatial abili-
ties, such as the ability to mentally rotate visual im-
ages and to retrieve spatio-temporal sequences, de-
cline with age (Iachini et al., 2008). Elderly peo-
ple also struggle more with multitasking, especially if
the tasks are complex (Wecker et al., 2005). Another
common age impairment is the decline in fluid intel-
ligence, which refers to the processing and reasoning
components of intelligence and the natural ability to
learn something new (Czaja and Lee, 2007). Because
of reduced processing efficiency, the working mem-
ory, which is the ability to keep information active
while processing or using it, declines with age (Salt-
house, 1991). Similarly, prospective memory, which
is the ability to remember to do something in the fu-
ture, also declines with age (Maylor, 1995). Another
issue emerging with age is the ability to select infor-
mation in the environment, e.g., to attend to informa-
tion on a Web page. Attention is the ability to focus on
a specific task or an object in the environment while
ignoring other things. This ability changes with age
and older people are slower to move their attention
from one thing to another (Czaja and Lee, 2007).
Another category of impairments due to aging is
that of changes in physical abilities. Response time
and accuracy of movement decline with age. Older
persons’ movements, reflexes, and reactions to stim-
uli are typically slower than younger persons’ (Farage
et al., 2012). Differently from the above-mentioned
normal age impairments, a disease that limits move-
ment is Parkinson’s, a neurodegenerative brain dis-
order that progresses slowly and worsens with age.
Symptoms are involuntary shaking of the hands, arms,
legs, jaw, chin, and lips. Other main symptoms in-
clude slowness of movement, stiffness of arms and
legs, and trouble with balance (Jankovic, 2008).
Even if not an impairment, another phenomenon
relevant for our work is anxiety towards technol-
ogy. In a study by Czaja and Sharit dating back to
1998, older people saw themselves as having less con-
trol over computers than younger people (Czaja and
Sharit, 1998). They had significantly less efficacy
in completing computer tasks, but, surprisingly, they
also perceived computers as being more useful com-
pared to younger people. Another study revealed that
older subjects reported higher levels of computer anx-
iety than younger persons and that the anxiety level
was related to the decision time on the computer when
performing a test (Laguna and Babcock, 1997). The
elderly may have a harder time using new technology,
but it helps with practice (Chu, 2010). These consid-
erations pushed us to develop a solution based on a
computing system but relying on a tangible interface
to ease user interaction.
Focusing on tangible user interfaces (TUIs),
they are intended to replace the graphical user inter-
faces (GUIs), more common in computing systems,
with real physical objects the user can interact with.
The key idea is to give digital information a physical
form and let these physical forms serve as a represen-
tation and control for digital information. A TUI lets
users manipulate digital information with their hands
and perceive it with their senses. One of the pioneers
in tangible user interfaces is Hiroshi Ishii, a profes-
sor at MIT who heads the Tangible Media Group at
the MIT Media Lab. His particular vision for TUIs,
called Tangible Bits, is to give physical form to digi-
tal information, making bits directly manipulable and
perceptible (Ishii, 2008b). Tangible Bits pursues the
seamless coupling between physical objects and vir-
tual data. TUIs will augment the real physical world
by coupling digital information to everyday physical
objects and environments (Ishii and Ullmer, 1997).
All physical objects can potentially be a part of a dig-
ital user interface (Ishii, 2008a). For example, if an
object, which is a part of a TUI, is moved or put in
a specific position, a digital signal will be sent from
either the tangible object itself or from another de-
vice that senses the object. Currently, there are dif-
ferent research areas and applications related to TUIs.
For instance, tangible augmented reality implies that
virtual objects are “attached” to physically manipu-
lated objects; in tangible tabletop interaction, physical
objects are moved upon a multi-touch surface; more-
over, physical objects can be used as ambient displays
or integrated inside embodied user interfaces.
TUIs have been used considerably in musical per-
formances and music-therapy treatments. A tangible
interface implying something “real”, “concrete”
offers a physical way to interact with music and sound
parameters. Most traditional musical instruments are
played through this kind of interaction, but the ad-
vent of digital technologies paved the way for innova-
tive and original approaches. Concerning technology-
enhanced TUIs for music, the scientific literature de-
scribes a number of theoretical approaches, proto-
types, and available products. Music TUIs can play
a number of roles: for example, synthesizers to gen-
erate sound, sequencers that perform audio samples
and mix them together, remote controllers for music
and sound parameters, or interfaces for music-related
CHIRA 2022 - 6th International Conference on Computer-Human Interaction Research and Applications
games. To cite but a few references, Paradiso et al.
review TUIs based on magnetic tags (Paradiso et al.,
2001), Newton-Dunn et al. describe a way to control
a dynamic polyrhythmic sequencer using physical ar-
tifacts (Newton-Dunn et al., 2003), and Schiettecatte
and Vanderdonckt present a distributed cube interface
based on interaction range for sound design (Schiette-
catte and Vanderdonckt, 2008). A noticeable example
of commercially available music TUI is the reacTable
a et al., 2007), used by renowned artists such as
ork in their live performances. Our solution adopts
a low-cost device called the Kibo, whose features will
be described in detail in Section 3.
Finally, let us address the applicability and role of
tangible user interfaces in rehabilitation. Motiva-
tion is one of the main problems evidenced in tradi-
tional therapy sessions, often hampered by the repet-
itive nature of exercises. Most studies show that an
effective rehabilitation must be early, intensive and
repetitive (Rego et al., 2014; Burke et al., 2010). As
such, these approaches are often considered repeti-
tive and boring by the patients, resulting in difficulties
in maintaining their interest and in assuring that they
complete the treatment program (Rego et al., 2014).
On the other hand, due to their nature, games can mo-
tivate and engage the patients’ attention and distract
them from their rehabilitation condition. On one side,
they require some motor and cognitive activity, but,
on the other, they have a story and can offer feed-
back and levels of challenge and difficulty that can
be adapted to the patients’ skills.
Serious games are an option that provides learn-
ing combined with entertainment. The locution “se-
rious games” refers to playful activities that provide
training and physical or mental exercise in a fun and
enjoyable way (D
orner et al., 2016). These games
can be not only a way to prevent the feeling of loneli-
ness (De Carvalho et al., 2012), but they can also en-
able social interaction (Fonseca et al., 2021). During
the last decades, digital games have become a popu-
lar leisure activity. Ijsselsteijn et al. claim that digital
games can be considered a promise to improve the
lives of seniors. To this end, it is important to develop
interesting and accessible games which could provide
an option to spend quality time with clear benefits
(Ijsselsteijn et al., 2007). Pearce reports that the stud-
ies of digital games with an emphasis on older peo-
ple are still rare because this need is relatively new
(Pearce, 2008). This perception may be justified by
the fact that the current elderly population did not
have much access to technology at their earlier age.
Many rehabilitation games based on TUIs are al-
ready available. For example, Handly is an integrated
upper-limb rehabilitation system for persons with a
neurological disorder (Vandermaesen et al., 2016).
Handly consists of tangibles for training four-hand
tasks with specific functional handgrips and a moti-
vational game. The system consists of four tangible
training boxes, which each present one essential grip
and associated hand task: push-pull, squeezing, knob
turning, and key turning. Handly combines tangibles
specifically designed for repetitive task-oriented mo-
tor skill training of typical daily activities with serious
gaming, thus offering a comprehensive approach.
Segara is an integrated hand rehabilitation sys-
tem for patients with rheumatoid arthritis (Zhao et al.,
2021). Segara consists of tangibles for training six
tasks with interactive functional handgrips and a mo-
tivational serious game. It shows that a system com-
bining games and tangibles to enhance hand rehabili-
tation is feasible and highly appreciated by patients.
Resonance is an interactive tabletop artwork that
targets upper-limb movement rehabilitation for pa-
tients with an acquired brain injury (Duckworth et al.,
2015). The artwork consists of several interactive
game environments, which enable artistic expres-
sion, exploration, and play. Resonance provides
uni-manual and bi-manual game-like tasks and ex-
ploratory creative environments of varying complex-
ity geared toward reaching, grasping, lifting, moving,
and placing tangible user interfaces on a tabletop dis-
play. Each environment aims to encourage collabora-
tive, cooperative, and competitive modes of interac-
tion for small groups of co-located participants.
NikVision is a tangible tabletop based on a user-
centered design approach for the cognitive stimula-
tion of older people with cognitive impairments and
dementia problems in nursing homes (Cerezo et al.,
2020). The general experiences of the users when
working with the tangible tabletop were assessed and
applied to the design of new cognitive and physical
stimulation activities. Game activities are specially
designed for the elderly, including daily tasks such
as getting dressed, cognitive assignments, and upper-
half motor skills training. Activities present different
levels of difficulty and audio feedback.
In this paper, we want to take advantage of real phys-
ical objects and the benefits that TUI technologies of-
fer in order to provide the elder with a playful way to
interact in rehabilitation therapies. As mentioned be-
fore, one of the advantages of a TUI is the user expe-
rience, because a physical interaction occurs between
the user and the interface itself. Another advantage is
usability since the user intuitively knows how to ma-
A Music Tangible User Interface for the Cognitive and Motor Rehabilitation of Elderly People
nipulate the interface by knowing the function of the
physical object; consequently, the user does not need
to learn the functionality. That is why TUIs are often
used to make technology more accessible for elderly
In this section, we will address the key technolo-
gies employed in the project: a music TUI called the
Kibo (Amico and Ludovico, 2020), the way it is con-
nected to a Web application, and the Web languages
and formats used to implement the browser games de-
scribed later.
The Kibo is a wooden board produced by Kodaly
that presents eight distinct, easy-to-recognize tangi-
These geometric shapes can be inserted into
and removed from the corresponding slots, thus trig-
gering events encoded in the form of MIDI messages.
The device is also sensitive to pressure variations on
single tangibles and individual dynamic responses can
be communicated via MIDI messages. As an addi-
tional controller, there is a knob that can be rotated
clockwise and counterclockwise. Finally, the device
is equipped with a gyroscope.
The Kibo can be connected via Bluetooth or USB
to iOS and macOS devices running a proprietary app
that acts as both a synthesizer and a configuration cen-
ter. Windows and Android operating systems are also
supported via third-party drivers. The communication
between the controller and the app occurs by exchang-
ing standard MIDI 1.0 messages. The MIDI engine
integrated into the app supports up to 7 Kibo units
simultaneously, without perceivable latency. This as-
pect is particularly interesting for collaborative expe-
riences (Barat
e et al., 2021). Being a fully compatible
controller, Kibo can also be integrated into any MIDI
setup without the intervention of the app as a media-
tor, as we will do in our proposal.
The control over music parameters is mainly
based on the 8 tangibles shown in Figure 1. Each ob-
ject has a different shape fitting in a single slot and
presents symmetry properties so that it can be rotated
and flipped before being inserted. Tangibles have a
magnetic core; they can be stacked one on top of
the other and interact through magnetic fields. The
body of the Kibo contains a multi-point pressure sen-
sor that allows for the detection of the insertion and
removal of tangibles. The characteristics of the sen-
sor make the instrument extremely sensitive and, si-
multaneously, very resistant.
Even if our proposal bypasses the native Kibos
app, it is interesting to analyze the three original op-
erating modes and shed some light on rehabilitative
and therapeutic scenarios:
Figure 1: The Kibos body and tangibles.
1. Musical Instrument Mode In this scenario,
Kibos tangibles are mapped onto pitches. As-
sociations between shapes and notes can be cus-
tomized, even triggering multiple notes through a
single tangible. The device is able to detect after-
touch effects, namely the possibility to track pres-
sure variations over tangibles after note attacks;
2. Beat Mode In this scenario, tangibles are
mapped onto single percussive instruments. The
pressure sensor, presenting a high level of resis-
tance to strong mechanical stresses but also a no-
ticeable sensitivity, allows effects ranging from
hard mallet beats to delicate brush rubbing;
3. Song Mode In this scenario, Kibo is employed
as a controller to trigger already available mu-
sic loops. Tangibles are associated with mutu-
ally synchronized but independent tracks, like in
a multi-track environment. When tangibles are
inserted, the corresponding tracks are activated;
when they are removed, tracks are muted.
The configurability of Kibo, coupled with the adop-
tion of standard communication protocols, enables
numerous and heterogeneous scenarios. Multiple
Kibo units in an ensemble can be configured to cover
distinct note ranges and timbres, or even to work in
different operating modes, thus providing the ther-
apist with great flexibility. Moreover, the standard
MIDI output of the Kibo allows the implementation
of additional operating modes where other meanings,
even extra-musical ones, can be assigned to user ges-
tures. Our proposal, described in detail in Section 4,
explores this possibility.
Even if originally conceived as a general-purpose
tangible MIDI controller, when used in a suitable sce-
nario the Kibo can also be considered an assistive
technology falling in the category of communication
boards. Under this perspective, it has a therapeutic
function since it encourages upper-limb movements
and challenges cognitive skills. Moreover, it is com-
pensatory both from a motor point of view, being able
to translate even small movements into sound, and
from a cognitive point of view, enabling intuitive mu-
sical expression by lowering the barriers of a tradi-
tional instrument.
CHIRA 2022 - 6th International Conference on Computer-Human Interaction Research and Applications
In this project, the Kibo has to be directly con-
nected via Bluetooth to the computing system that
hosts the browser. A specific protocol called MIDI
over Bluetooth or, simply, Bluetooth MIDI is spe-
cially conceived to exchange MIDI messages over
Bluetooth connections (Bartolomeu et al., 2005). The
Kibo adopts Bluetooth Low Energy (BLE), a wireless
personal-area network technology that, compared to
the original Bluetooth protocol, is intended to provide
considerably reduced power consumption and cost
while maintaining a similar communication range.
A BLE MIDI device will transparently operate with
a MIDI-compatible application on most mobile and
desktop platforms with no additions and provides “out
of the box” support for most use cases compared with
any wired or wireless alternative.
The visual feedback of the system is imple-
mented in the form of a Web page by adopting W3C
standard languages and formats. As a result, the
browser games can be experienced using any HTML5
browser. The project was realized using HTML, CSS,
and JavaScript.
Please note that all the mentioned technologies are
client-side, so the platform could also be enjoyed lo-
cally on the user’s client, with no need to connect to
a server. Nevertheless, we have publicly released it
over the Web in order to distribute it easily at no cost
for the user and to keep it up-to-date in the case of a
new release.
The goal of linking music therapy, rehabilitation ex-
ercises, and technology through the Kibo brought to
the design and implementation of a Web platform that
proposes three serious games. The name of the plat-
form is Kibo Web Games. All game activities allow
working the association between physical elements
(geometric tangibles) and the concepts they represent
(e.g., notes, tracks, hit buttons). The platform is avail-
able at https://kibogames.lim.di.unimi.it/ and its orig-
inal Web interface is shown in Figure 2. Kibo Web
Games need a Kibo device connected via BLE.
A number of game parameters have been intro-
duced to allow flexibility in usage and adaptation to
gradual improvements without causing frustration in
the player. The user or therapist can set different types
of activities and levels of difficulty according to the
following parameters: i) the game to play, that is basi-
cally a rhythm-based activity where given shapes are
proposed to the player and must be timely touched,
inserted, removed, or continuously pressed depend-
ing on the game mode, as explained in the following;
Figure 2: The interface of Kibo Web Games.
ii) the number of shapes (1 to 8 out of the 8 avail-
able) that can randomly be involved during the game
experience; iii) the total length of the game sequence,
i.e. the number of actions a user should carry out in a
game session; and iv) the speed of the shape genera-
tion and scroll animation.
The central part of the interface is taken by the
Game field, which displays the moving shapes during
a game session. Shapes move from right to left, and
the perfect timing for user’s actions is when they hit
the black vertical line. Another relevant part of the
interface in Figure 2 is the MIDI message console,
namely the lower rectangle that displays the MIDI
messages received from the Kibo. Finally, the Play
button starts the selected game with the parameters
set by the user.
The second game expects the user to insert or re-
move the shapes from their slots in synchronization
to the game’s graphic scenario is able to train preci-
sion in performing movements. The third game asks
the user to simultaneously press and release the rec-
ognized shapes. The scores, levels of difficulty, and
feedback given to the player depend on her/his tem-
poral precision in performing the movement.
Concerning the gameplay, Kibo Web Games focus
on the interactions occurring between the player and
the Kibo, i.e. simple motor movements like hitting,
tapping, grabbing, holding, releasing, placing, and
removing the geometrical tangibles. Furthermore,
recognition and listening skills are trained to help re-
store or keep active cognitive functions. The Kibo
gives tactile and musical feedback to every action that
is performed and allows the user to proceed in small
steps. It also allows users to manipulate objects giving
the clearer image of the connection between physical
interaction and the response that it triggers.
All games share the same game field and func-
tional concepts. The system generates shapes
scrolling across the screen from right to left and the
A Music Tangible User Interface for the Cognitive and Motor Rehabilitation of Elderly People
goal is to timely recognize the shape by hitting, press-
ing, inserting, or removing the corresponding tangible
from the Kibo physical body when it reaches the ver-
tical line in the delimiter box (Figure 2). Throughout
the duration of a game session, the arrow, square, and
vertical line in the Game field animate to facilitate
the game experience. Arrow and square flash when
the user is supposed to perform a task. The vertical
line turns either red or green according to the accu-
racy of the performance. The score field keeps track
of the points gained while playing and remains dis-
played until the next game session is started.
Perfect hits are those performed in a very small
range around the computed timing. Good hits occur
in a slightly wider timing range. The exact values de-
pend on the game speed set by the user, but they are
in the order of tenths of a second. Currently, score
penalties for wrong shape recognition or missed hits
are not implemented to avoid frustration in players,
but this feature could be easily integrated.
Please note that the Kibo Web Games interface
allows users to play with the Kibo instrument as
a simple musical controller, regardless a game ses-
sion is active or not, after connecting it to the Web
page. This provides sound feedback (C-major scale)
and console messages only, but it might help to gain
confidence and increase motivation. As another re-
mark, the three games help gain rhythm awareness
and present musical feedback, but they never imply
a music-education background nor make use of any
kind of music-related concept.
Game A Tap the Shapes is based on the Kibos
Beat Mode (see Section 3). Tangibles are mapped
onto single notes of a C-major scale, one grade per
shape. Pressing, hitting, or tapping a shape sends a
Note On message immediately followed by a Note
Off. MIDI messages are interpreted by the interface
and played back thanks to an embedded audio syn-
thesizer. Difficulty varies according to speed, number
of shapes involved, game length, and score precision.
The rhythm game expects the user to simply activate
a midi ON message by pressing the expected tangible
in the right timing range, namely when the arrow and
delimiter box turn yellow. In order to play this game,
all tangibles must be inserted.
Game B Insert and Remove Shapes recalls the
Kibos Song Mode (see Section 3). Tangibles are as-
sociated with independent synchronized tracks within
a multi-track environment realized with Pro Tools
for this purpose. The multi-track starts playing when
the game is initialized but has no volume. Each track
is associated with a shape and is unmuted when the
corresponding tangible is inserted into the device’s
wooden base. Conversely, removing the shape will
mute the track. All tangibles must be removed and
placed in front of the player or at a reachable distance
before starting the game. For a better user experience,
long game sequences must be set. The aim of Game B
is to score points by inserting or extracting the Kibos
tangibles once the randomly generated shapes touch
the hit-line or perfectly fit the delimiter box before the
animation ends and the shape disappears. The random
algorithm should assure that, in the case of right user
performance, at the end of the session no shapes are
left inside Kibos body. This game mode solicits the
development or recovery of some cognitive abilities,
including memory (shapes have to be quickly recog-
nized, found in the space around the Kibo, and in-
serted into the right slot) and creative reasoning (e.g.,
finding the best initial layout for the pieces around the
Kibos body).
Game C Hold and Release Shapes recalls the
Kibos Musical Instrument Mode and takes benefits
from the polyphonic aftertouch feature. Tangibles
are mapped once again onto the grades of a C-major
scale. Somehow similar to Game A, when a shape
touches the vertical line or enters the delimiter box,
the user is expected not only to tap the correspond-
ing tangible but also to keep it pressed until the same
shape appears again. Pressing a tangible sends a
MIDI Note On message, and releasing it sends the
corresponding Note Off message. This game mode is
probably the most challenging one, from both a mo-
tor and a cognitive point of view. For example, some
combinations of shapes require not only the ability
to have them selected simultaneously but also a good
strategy to have a hand free for the next insertion.
Due to the restrictions imposed on society during the
COVID-19 pandemic and the impact it had on the el-
derly and fragile populations, it has not been possi-
ble to test this prototype in a real-life setting so far.
Nonetheless, the games were proposed to elder fam-
ily members, colleagues, and domain experts to get
objective feedback on their benefits and usefulness.
Overall, Kibo Web Games were appreciated and
described as an enjoyable form of entertainment, yet
challenging enough to possess rehabilitative proper-
ties. Differentiating between similarly shaped tangi-
bles (e.g., star and flower) was initially perceived as
demanding by many but overcome with practice. An-
other challenging aspect was maintaining the level of
performance with the speed increment.
An early prototype was presented to music thera-
CHIRA 2022 - 6th International Conference on Computer-Human Interaction Research and Applications
pists and physiatrists of Fondazione Don Carlo Gnoc-
chi, Milan. Many useful remarks emerged about the
games’ structure and the user interface. Game A was
particularly appreciated due to its simplicity, but the
interface was considered too complex for users with
impairments, above all cognitive ones. A first sugges-
tion was to move the parameter configuration and the
MIDI console away from the game screen, thus intro-
ducing a sort of back-office area to set game sessions;
in fact, in a standard scenario, the parameter config-
uration on one side and the gameplay on the other
involve two distinct actors, and the presence of side
controls and messages can be a source of distraction.
Similarly, the use of colors should be limited as much
as possible, just to differentiate the actions required
(e.g., either insert or remove a tangible). The nu-
meric score, too, should be turned into a more direct
and comprehensible representation, such as a given
number of stars, a growing bar, or something similar.
Games B and C involve persistent actions (i.e. shapes
to keep inserted or pressed for a given time). In this
case, the experts’ objections focused on the graphi-
cal representation itself of the game field. In fact, in
their opinion, having 8 scrolling areas corresponding
to the 8 tangibles could help, above all in the case of
cognitive impairment. Actually, a colored multi-track
representation for Game C would recall the piano-roll
visualization typically adopted by MIDI sequencers.
Moreover, experts agreed that the last two games are
more challenging from any point of view and, for this
reason, they suggested a step-by-step process before
playing the final version (e.g., first locate given shapes
with no time constraint, then test the physical actions
by moving shapes into their slots, etc.). Early experi-
mentation of the revised interface should start at Fon-
dazione Don Carlo Gnocchi in October 2022. Thanks
to the positive feedback received, it is likely that Kibo
Web Games have the requisites to be utilized among
the elderly population. Future testing on this specific
population will be necessary to fine-tune the develop-
ment of a truly suitable device that can be used in the
rehabilitative space. Connecting performance results
to a database or exporting them into a CSV file could
provide a more accurate track of progress.
The approach of Kibo Web Games can be easily
generalized and adapted to other scenarios. For ex-
ample, the platform can be profitably used to develop
music parameters and geometric concepts awareness
in young students, even in preschool age, thanks to
TUI-based playful activities. Moreover, Kibo Web
Games can help recover cognitive and motor abilities
not only in elderly people but also in younger users;
some experiences in this sense are reported in (Barat
et al., 2021).
In this project, we investigated the suitability of the
Kibo to compensate for age impairments. The device
proved to be a good solution from a technical and a
physical point of view due to user-friendliness and ro-
bustness. Unfortunately, it is not an affordable prod-
uct. Thanks to the Kibos 8-voice polyphony, mul-
tiple participants have the opportunity to access the
interface simultaneously. Collaboration could be ad-
vantageous for seriously disabled individuals needing
assistance from caregivers. However, the described
games are not designed to explore this scenario in full.
The possibility to choose parameters and set dif-
ficulty levels allows users to progress at their own
pace without confronting insurmountable obstacles.
A therapist or a caregiver can suitably configure game
sessions based on the players’ attitudes and abilities.
Moreover, no game penalties were implemented with
the distinct purpose of not discouraging elderly users
who are often not technology savvy and likely to be
afraid of making mistakes or getting stuck.
The inspiration for future work mainly comes
from the observations and remarks by experts re-
ported in Section 5. Short-term future improvements
include the creation of a different game field for each
specific game, keeping the interface’s look as sim-
ple as possible to avoid any distractions during the
gameplay, and the implementation of two different
windows (one for the user/patient who can only play
the game and one for the specialist who can set the
game parameters). In addition, tracking parameters
like pressure variations could test and improve tac-
tile and fine motor skills in action. After on-the-field
experimentation, gaming speed will probably need to
be adjusted to accommodate slower apprehension in
aged people. Finally, an aspect to further explore is
the use of proper musical feedback to be applied in
music therapy.
Our expectation is that Kibo Web Games are able
to stimulate declining cognitive abilities and reignite
strained motor skills in elderly people in order to keep
them active and fully integrated into society through
a set of engaging activities.
Amico, M. D. and Ludovico, L. A. (2020). Kibo: A MIDI
controller with a tangible user interface for music ed-
ucation. In Int. Conf. on Computer Supported Educa-
tion, pages 613–619. SCITEPRESS.
e, A., Ludovico, L. A., and Oriolo, E. (2021). An
ensemble of tangible user interfaces to foster mu-
sic awareness and interaction in vulnerable learn-
A Music Tangible User Interface for the Cognitive and Motor Rehabilitation of Elderly People
ers. In Proc. 5th Int. Conf. on Computer-Human In-
teraction Research and Applications, pages 48–57.
Bartolomeu, P., Fonseca, J., Duarte, P., Rodrigues, P., and
Girao, L. (2005). Midi over bluetooth. In 2005 IEEE
Conference on Emerging Technologies and Factory
Automation, volume 1, pages 8 pp.–102.
Burke, J., McNeill, M., Charles, D., Morrow, P., Crosbie,
J., and McDonough, S. (2010). Augmented reality
games for upper-limb stroke rehabilitation. In 2010
Second International Conference on Games and Vir-
tual Worlds for Serious Applications, pages 75–78.
Cerezo, E., Bonillo, C., and Baldassarri, S. (2020). Ther-
apeutic activities for elderly people based on tangible
interaction. In ICT4AWE, pages 281–290.
Chu, R. J.-c. (2010). How family support and internet
self-efficacy influence the effects of e-learning among
higher aged adults–analyses of gender and age differ-
ences. Computers & Education, 55(1):255–264.
Czaja, S. J. and Lee, C. C. (2007). The impact of aging on
access to technology. Universal access in the infor-
mation society, 5(4):341–349.
Czaja, S. J. and Sharit, J. (1998). Age differences in atti-
tudes toward computers. The Journals of Gerontology
Series B: Psychological Sciences and Social Sciences,
De Carvalho, R. N. S., Ishitani, L., Nogueira Sales De Car-
valho, R., et al. (2012). Motivational factors for mo-
bile serious games for elderly users. Proceedings of
XI SB games, pages 2–4.
orner, R., G
obel, S., Effelsberg, W., and Wiemeyer, J.
(2016). Serious games. Springer.
Duckworth, J., Mumford, N., Caeyenberghs, K., Eldridge,
R., Mayson, S., Thomas, P. R., Shum, D., Williams,
G., and Wilson, P. H. (2015). Resonance: an in-
teractive tabletop artwork for co-located group reha-
bilitation and play. In Int. Conf. on Universal Ac-
cess in Human-Computer Interaction, pages 420–431.
Farage, M. A., Miller, K. W., Ajayi, F., and Hutchins,
D. (2012). Design principles to accommodate older
adults. Global journal of health science, 4(2):2.
Fonseca, X., Slingerland, G., Lukosch, S., and Brazier, F.
(2021). Designing for meaningful social interaction
in digital serious games. Entertainment Computing,
Iachini, T., Borghi, A. M., and Senese, V. P. (2008). Cat-
egorization and sensorimotor interaction with objects.
Brain and Cognition, 67(1):31–43.
Ijsselsteijn, W., Nap, H. H., de Kort, Y., and Poels, K.
(2007). Digital game design for elderly users. In
Proceedings of the 2007 Conference on Future Play,
Future Play ’07, page 17–22, New York, NY, USA.
Association for Computing Machinery.
Ishii, H. (2008a). Tangible bits: Beyond pixels. In Proc.
2nd Int. Conf. on Tangible and Embedded Interac-
tion, TEI ’08, page xv–xxv. Association for Comput-
ing Machinery.
Ishii, H. (2008b). The tangible user interface and its evolu-
tion. Commun. ACM, 51(6):32–36.
Ishii, H. and Ullmer, B. (1997). Tangible bits: towards
seamless interfaces between people, bits and atoms.
In Proceedings of the ACM SIGCHI Conference on
Human factors in computing systems, pages 234–241.
Iversen, T. R. (2015). Exploring tangible interaction: Alter-
native interfaces for assisting elderly users. Master’s
thesis, Department of Informatics, University of Oslo.
Jankovic, J. (2008). Parkinson’s disease: clinical features
and diagnosis. Journal of neurology, neurosurgery &
psychiatry, 79(4):368–376.
a, S., Geiger, G., Alonso, M., and Kaltenbrunner, M.
(2007). The reacTable: exploring the synergy be-
tween live music performance and tabletop tangible
interfaces. In Proc. 1st Int. Conf. on Tangible and Em-
bedded Interaction, pages 139–146.
Laguna, K. and Babcock, R. L. (1997). Computer anxiety
in young and older adults: Implications for human-
computer interactions in older populations. Comput-
ers in human behavior, 13(3):317–326.
Maylor, E. A. (1995). Prospective memory in normal ageing
and dementia. Neurocase, 1(3):285–289.
Newton-Dunn, H., Nakano, H., and Gibson, J. (2003).
Block jam: A tangible interface for interactive music.
Journal of New Music Research, 32(4):383–393.
Paradiso, J. A., Hsiao, K.-y., and Benbasat, A. (2001). Tan-
gible music interfaces using passive magnetic tags. In
Proceedings of the 2001 Conference on New Inter-
faces for Musical Expression, NIME ’01, page 1–4,
SGP. National University of Singapore.
Pearce, C. (2008). The truth about baby boomer gamers:
A study of over-forty computer game players. Games
and Culture, 3(2):142–174.
Rego, P. A., Moreira, P. M., and Reis, L. P. (2014). Archi-
tecture for serious games in health rehabilitation. In
New Perspectives in Information Systems and Tech-
nologies, volume 2, pages 307–317, Cham. Springer.
Salthouse, T. A. (1991). Mediation of adult age differ-
ences in cognition by reductions in working mem-
ory and speed of processing. Psychological Science,
Schiettecatte, B. and Vanderdonckt, J. (2008). Audiocubes:
A distributed cube tangible interface based on interac-
tion range for sound design. In Proc. 2nd Int. Conf.
on Tangible and Embedded Interaction, TEI ’08, page
3–10, New York, NY, USA. Association for Comput-
ing Machinery.
Vandermaesen, M., De Weyer, T., Feys, P., Luyten, K., and
Coninx, K. (2016). Integrating serious games and tan-
gible objects for functional handgrip training: A user
study of handly in persons with multiple sclerosis. In
Proc. of the 2016 ACM Conf. on Designing Interactive
Systems, pages 924–935.
Wecker, N. S., Kramer, J. H., Hallam, B. J., and Delis, D. C.
(2005). Mental flexibility: age effects on switching.
Neuropsychology, 19(3):345.
Zhao, X., Deng, Z., Jin, M., Li, X., Zhang, W., Jin, M.,
Ying, F., Wang, Q., and Wang, G. (2021). Segara:
Integrating serious games and handgrip for hand reha-
bilitation in rheumatoid arthritis patients. In The 9th
Int. Symp. of Chinese CHI, pages 101–104.
CHIRA 2022 - 6th International Conference on Computer-Human Interaction Research and Applications