THE MUSIC PAINT MACHINE
A Multimodal Interactive Platform
to Stimulate Musical Creativity in Instrumental Practice
Luc Nijs, Pieter Coussement, Chris Muller, Micheline Lesaffre and Marc Leman
Institute for Psychoacoustics and Electronic Music, Ghent University, Blandijnberg 2, 9000 Ghent, Belgium
Keywords: Music Education, Multimodal Interactive Platform, Musical Creativity, Improvisation, Embodied Music
Cognition.
Abstract: Interactive music systems offer new possibilities to support instrumental music teaching by providing a
corporeally grounded experience as a basis for understanding music and music playing. In this paper we
introduce the Music Paint Machine, a device that enables music performers to make a painting on a
computer screen by playing their instrument. It is hypothesized that using this application stimulates
understanding and creative use of musical parameters.
1 INTRODUCTION
In music education and more particular in
instrumental studio teaching, the use of computers to
support the teaching and learning process is growing
but is nevertheless still in a rather early stage.
Computers are used to make and analyse recordings
(e.g. intonation, mistakes), the World Wide Web is
used to provide information and to communicate
outside the lessons (e.g. website, electronic learning
environments), and software is developed for student
assessment. Furthermore the use of music notation
software is widely spread in music education. An
important and recent development is the design and
implementation of tools that support the learning
process by measuring the instrumental gestures and
posture of for example string players (e.g. KC Ng et
al., 2007) and clarinettists (e.g. Wanderley, Vines,
Middleton, McKay, & Hatch, 2005).
Almost nonexistent in instrumental music
teaching is the educational use of interactive music
systems. Although a variety of these technological
applications have been used for quite some time in
performances, they have not yet found their way into
the domain of musical instrument teaching.
Interactive music systems could, however, address
current issues regarding motivation and creativity in
music education (Bamford, 2007), by providing an
experiential basis for learning in which methodically
designed learning paths can be combined with more
exploratory ways of learning to play a musical
instrument .
In this paper we introduce such an educational
interactive music system, the Music Paint Machine.
2 THEORETICAL FRAMEWORK
The theoretical background of the conceptual design
of the Music Paint Machine is the Embodied Music
Cognition paradigm. This research paradigm
acknowledges the embodied nature of the musical
mind (Leman, 2007). What happens in the mind
depends on properties of the body and therefore
body and body movement have an impact on
meaning formation.
An important aspect of embodied music
cognition is the multimodal nature of musical
involvement and expression. Visual, auditory and
haptic/proprioceptive perception can strongly
interact. One modality can, for instance,
disambiguate information in another modality;
different modalities can provide a means of
calibration for one another and a percept from one
modality can even override that of another modality
(Ernst & Bülthoff, 2004). Most importantly, these
interactions are mostly unconscious and they are
spontaneous. Music can therefore not be studied as
merely sound (Schutz, 2008); it should be studied as
a multimodal phenomenon.
331
Nijs L., Coussement P., Muller C., Lesaffre M. and Leman M. (2010).
THE MUSIC PAINT MACHINE - A Multimodal Interactive Platform to Stimulate Musical Creativity in Instrumental Practice .
In Proceedings of the 2nd International Conference on Computer Supported Education, pages 331-336
DOI: 10.5220/0002859103310336
Copyright
c
 SciTePress

Interactive music systems use sensing technologies
and software applications that enable users to
explore and creatively exploit the multimodal nature
of corporeal intentions and expressive articulations,
while being engaged in music. Accordingly,
interactive music systems facilitate gestural and
multimodal involvement with music. Therefore their
use in instrumental teaching can contribute to a more
embodied approach to instrumental music education
as opposed to a prevailing cognitive-emotional
approach. By using applications that stimulate full
corporeal engagement in music, students learn to
understand the bodily basis of musical meaning.
3 THE MUSIC PAINT MACHINE
3.1 Concept
The Music Paint Machine is an interactive music
system that introduces movement and
experimentation in musical instrument teaching. It
allows music performers to make a painting by
simultaneously moving the body and playing music.
The painting takes shape by combining musical
features (i.e. pitch, loudness, playing style) and body
movement (i.e. moving the upper body, trigger
sensors with feet on a dance mat).
3.1.1 Learning Goals
1. Stimulate Creativity Through Playfulness with
Musical Parameters
An important element of musical creativity is
playfulness with musical parameters (Deliège &
Wiggins, 2006; Sloboda, 2000). The combination of
playing, moving & painting leads to a better
understanding of these parameters by providing a
corporeally grounded experience as an experiential
basis for learning and by stimulating students to
experiment with these parameters. The introduction
of visual feedback makes users forget the
technicalities of playing their instrument and affords
them to immerse in an action-perception loop and
intuitively respond to what happens on the screen. In
this way the Music Paint Machine induces a shift
from “controlling” to “experiencing” musical
parameters.
2. Develop Confidence and Skills to Improvise
Everybody can paint or draw something. The Music
Paint Machine offers the opportunity to learn to “let
go” and play something from scratch by translating
musical improvisation into a more familiar thing to
do, i.e. painting or drawing. It even can become a
challenge to paint complex, beautiful or funny
pictures by playing the musical instrument. In this
way students can gain confidence and audacity to
improvise.
Besides lowering the threshold to improvise, the
Music Paint Machine can also be used to further
develop improvisation skills. According to Welch
and Adams (2003) improvisation in music is
fostered through encouraging the learning of basic
musical elements. Improvisation skills are based on
building blocks of simple musical behaviours that
should be practiced. The development of these skills
starts with exploration of all the possibilities to
create sound (Kratus, 1991; Scott, 2007). The Music
Paint Machine provides an excellent way to
encourage musical exploration but also to help a
student developing throughout the different levels of
improvisation and thus acquiring the skills to
creatively improvise (Kratus, 1991).
3. Develop Embodied Musicianship
Musicianship is in essence procedural knowledge,
rooted in practice and invariably embodied
(Bowman, 2000; Elliott, 1995). An important aspect
of musicianship is listenership (Elliott, 1995).
Listening to music (both when performing or not) is
a “hearing-as”, the foundation of which is the body
(Bowman, 2004). What we hear is translated into a
so-called action oriented ontology, i.e. a repertoire
of movements that are sedimented in our body
schema through an action-perception coupling and
provide a reference against which musical meanings
can be attributed (Bowman, 2004; Leman, 2007).
Listening and musical understanding therefore can
be refined trough the use of body movements
(Pierce, 2007). Therefore, acknowledging the
embodied nature of music cognition means
acknowledging the bodily basis of musical
knowledge and understanding.
The Musical Paint Machine integrates body
movements in its mapping. By using the body to
play and paint, music students can explore the
possibilities of their instrument and experiment with
movement and musical parameters. Through specific
drawing tasks, movements can be used to elicit
bodily understanding of certain musical elements
such as phrasing, dynamics and articulation
(transition between notes or not).
Because the Music Paint Machine offers the
possibility to represent movement and sound in a
common visual stimulus, it accommodates the
multimodal nature of musical expression and
CSEDU 2010 - 2nd International Conference on Computer Supported Education
332

involvement. By combining movement, sound and
visuals, it can turn learning to play music into a
“gesamt-erfahrung”, i.e. a multimedia event in
which different forms of artistic expression are
combined and lead to a multimedia output. Through
a creative and playful use of musical parameters,
based on the integration of sound, movement and
visuals, a proficient user can turn this output into an
artistic creation, a kind of twenty-first century
“gesamtkunstwerk”.
3.1.2 Didactic benefits
1. A Tool for Pedagogic Documentation
Pedagogical documentation is a tool for
participatory and formative evaluation (Dahlberg,
Moss, & Pence, 1999; MacDonald, 2007). It aims at
visualizing and understanding what is going on
during a lesson and what the child is capable of
without any predetermined framework of
expectations and norms.
In view of its pedagogic goals, the Music Paint
Machine can be used as a tool for reflective
discussion by using the students’ artistic creations as
pedagogic documentation. The comparison between
drawings and music can reveal different aspects of
the student’ s playing (e.g. how creative a student
deals with musical parameters) and of the learning
process. Not only is it possible to compare different
paintings over a certain period of time, but thanks to
the software that is implemented in the Music Paint
Machine, it is also possible to (re)view each play
session in different representation modes, by
including or excluding the time dimension and by
changing the view between different angles (see
section 3.2.2). This enables teacher and student to
discuss features of music playing such as amount of
movement (e.g. use of many colours indicating a lot
of movement with the feet), ways of moving (e.g.
more vertical then horizontal), correlations between
sound, visuals and movement and the like. In this
way the Music Paint Machine contributes to the
development of bodily awareness in the context of
embodied music cognition.
2. A Tool for Student Assessment
The Music Paint Machine can be regarded as an
“artistic” measurement tool, complementing an
analysis based on objective measurement data.
However, the fact that it provides both, allows one to
compare artistic output and objective measurement.
This comparison might reveal interesting aspects of
the students learning process by revealing the link
between product and process.
Furthermore, students can store their creations
and include them in a portfolio. Or they can be put
on the wall of the classroom. This enables peer
evaluation which is an underestimated but very
valuable component of an efficient evaluation
system (Nijs, 2008). In this way it is possible to map
the progress of the student as visualized in his
artistic creations.
3. A Motivator
The Music Paint Machine contributes to the
development of intrinsic motivation to play the
instrument by increasing the fun factor of
instrumental music lessons or practice sessions at
home. It provides an opportunity to part with the
schoolish character of musical instrument lessons.
It can be used to offer challenges that can easily
be adapted to match the skills of the player.
Moreover, by introducing body movements into the
game, it addresses the whole body thereby
enhancing the corporeality of the experience. This
has a major influence on the probability of having a
flow experience. The Music Paint Machine enables
events that increase the body’s capacity to act while
playing music and thereby it can bring joyful
experiences (Massumi, 2002).
3.2 Description of the System
3.2.1 Hardware
Colour Dance Mat
The hardware of the multicolored dance mat consists
of 12 pressure sensors (contact switches), 4 extra
switches (situated on top of the mat) and a USB
interface which are all integrated in the MDF floor
plate. A cover presenting a twelve-colour wheel
hides the hardware and makes choosing colours in
the game clear. Stepping on a colour activates a
pressure sensor underneath. The USB interface is a
hacked numeric keypad. The twelve contact
switches replace the button switches of the original
keyboard matrix.
Motion Sensors
The movements of the user are captured by a Wii
motion plus attached to the torso, with a flexible
strap that doesn’t hinder breathing.
THE MUSIC PAINT MACHINE - A Multimodal Interactive Platform to Stimulate Musical Creativity in Instrumental
Practice
333

Figure 1: overview of the system.
3.2.2 Mapping and Feature Extraction
The software for The Music Painting Machine is
developed in MAX/MSP and the visualisation is
done in jitter, using the integrated OpenGL engine.
Mapping Body Movement
The canvas that is either portrayed on a display or
projected on some surface, reflects the player’s
movement and choices of colour. The pressure
sensors, embedded in the dance mat, give the player
the opportunity to choose the drawing colour. A set
of 12 basic colours is available and this initial
colour’s saturation can be dynamically controlled by
moving the torso either forward (more saturated) or
backwards (less saturated). The movement of the
torso also determines the X-position of the
paintbrush on the screen. The Wii remote, strapped
to the chest, is used to capture leaning forward and
backward (pitch) and turn left or right (roll).
Mapping Musical Features
All other drawing commands are determined by
musical features. The vertical position of the
paintbrush on the canvas is determined by pitch. A
sustained note produces a horizontal line, while a
melody produces a curved line that follows the
melodic contour. The thickness of the paintbrush is
determined by the loudness of what is played. The
louder a user plays, the thicker the brushstroke
becomes.
Loudness and pitch are currently tracked by an
object of the CPS programming environment (see:
cps.bonneville.nl).
Presentation Modes: Visualizing the time Dimension
As already mentioned, the X-position and Y-position
of the painting brush are determined by feature
extraction, leaving the Z-axis untouched. In the
Music Paint Machine, this Z-axis represents the time
you spend playing. While playing, the player is
presented with a two dimensional view, without any
indication of time, giving him the opportunity to
explore his musical painting to the fullest. When
he’s done playing, one of the additional top switches
on the dance mat enables the player to change his
viewpoint, to gain access to the third dimension,
representing time that is plotted on the Z-axis. For
instance, rotating the presented image by 90 degrees
over the Y-axis reveals a coloured melodic contour
over time. This kind of representation can be used to
further analyse the performance. One of the other
available representation modes is an animated replay
of what is drawn, either playing alongside the music
or with the music muted.
4 DISCUSSION
4.1 Sound and Images
Interactions between auditory and visual processing
can occur regardless of the level of relevancy
between them (Hidaka et al., 2009), so presenting
congruent visual feedback maximises the
possibilities of visual feedback augmenting the
process of musical creation. Despite growing
evidence that supports the effectiveness of
visualization as a didactic tool, traditional
instrumental music teachers remain sceptical
towards the integration of visual feedback. They
often argue that visual feedback interferes with
listening and with learning to audiate (i.e. the ability
to hear and comprehend in one’s mind the sound of
music that is not or may never have been physically
present (Gordon, 1997). Existing objections against
the use of visual feedback are grounded in some
misconceptions or a misunderstanding of the nature
of music and musical understanding. Besides
interfering with one another, visual and auditory
stimuli can reinforce each other (Ernst & Bülthoff,
2004; Lipscomb, 2005). Their combination can
enhance learning processes (e.g. Forsythe & Kelly,
1989; Rogers, 1991) and musical experience (e.g.
Davidson, 1993; Frego, 1999). This is in line with
findings that the connection between auditory, visual
and tactile stimuli is essential for the development of
musical perception (Gembris, 2006).
CSEDU 2010 - 2nd International Conference on Computer Supported Education
334

Therefore we believe the Music Paint Machine in
many ways can fulfil a complementary role to
existing systems or didactic methods. Firstly, it
complements the traditional use of visual feedback
that most often limited to the score and personal
annotations in it (e.g. to mark important passages, to
stress expressive features). Secondly, by providing a
combination of artistic visualization and objective
data, the Music Paint Machine complements existing
educational applications that are based on
visualizing objective data (Bevilacqua, Guédy,
Schnell, Fléty, & Leroy, 2007; Ng, Larkin,
Koerselman, & Ong, 2007). As has been explained
in this article, this is valuable for music education.
But it also has an important benefit for using it as a
research tool.
4.2 A Tool for Research
The Music Paint Machine is an application that is
tailored to the embodied music cognition research
paradigm. It enables to investigate tool related
experiences (shift from subject to user), it can easily
be used in a classroom or at home (shift from lab to
ecological setting) and when used in a classroom
setting it will reveal aspects of the role of social
interaction with teachers and peers (shift from
individual experience to social interaction). An
important aspect of the Music Paint Machine that
contributes to the ecological validity of experiments
in which it is used, is its potential to engage users in
a strong sensation of immersion and make them
forget they are doing an experiment. Moreover, due
to a focus on artistic creation by playing and
moving, users do not have the impression of being
measured and analysed, which can lead to non-
representative measurements. What appears on
screen is nor a visualization of objective data, nor an
exact capturing of movements and posture. It is, on
the contrary, a creative output that appeals to
imagination.
Because of its combination of artistic and
objective measurement data, the Music Paint
Machine also contributes to the expansion of
methods that accompanies the aforementioned
paradigm shift. It enables the combination of
subjective and objective measurement through the
implementation of state of the art monitoring
technologies. Furthermore it deals with the transfer
between modalities. Experiments with the Music
Paint Machine can contribute to existing research on
cross-modality and the precise coupling of different
modalities (Naveda & Leman, 2009).

5 CONCLUSIONS AND FUTURE
WORK
In this paper we have outlined the theoretical
framework and the conceptual design of the Musical
Paint Machine, an interactive music system that
enables students to create real-time visualizations of
the music they play. Furthermore we discussed
possible didactic benefits and our expectations
regarding the use of this application.
From the theoretical point of view, this interactive
music systems holds promising potential. Of course,
empirical validation of the theoretical elaboration is
necessary. In the near future we start a series of
experiments that probe the users’ experience. Based
on these experiments and on a close collaboration
with instrumental music teachers, specific tasks will
be designed for a series of experiments that test the
didactic efficacy of the Music Paint Machine.
ACKNOWLEDGEMENTS
This work is funded by the EmcoMetecca project of
Ghent University. We want to thank Ivan Schepers
for the realization of the colour dance mat.
REFERENCES
Bamford, A. (2007). Kwaliteit en consistentie. Arts and
cultural education in Flanders: CANON Cultural
Unit.
Bevilacqua, F., Guédy, F., Schnell, N., Fléty, E., & Leroy,
N. (2007). Wireless sensor interface and gesture-
follower for music pedagogy. Paper presented at the
7th international conference on New interfaces for
musical expression, New York.
Bowman, W. (2000). A somatic," here and now" semantic:
Music, body, and self. Bulletin of the Council for
Research in Music Education(144), 45-60.
Bowman, W. (2004). Cognition and the body:
Perspectives from music education. In L. Bresler
(Ed.), Knowing Bodies, Moving Minds: Towards
Embodied Teaching and Learning (pp. 29). Dordrecht,
The Netherlands: Kluwer Academic Publishers.
Dahlberg, G., Moss, P., & Pence, A. (1999). Beyond
quality in early childhood education and care:
Postmodern perspectives. London: Routledge.
Davidson, J. (1993). Visual perception of performance
manner in the movements of solo musicians.
Psychology of Music, 21(2), 103.
Deliège, I., & Wiggins, G. (2006). Musical creativity:
multidisciplinary research in theory and practice.
New York: Psychology Press.
THE MUSIC PAINT MACHINE - A Multimodal Interactive Platform to Stimulate Musical Creativity in Instrumental
Practice
335

Elliott, D. (1995). Music matters. A new philosophy of
music education. New York: Oxford University Press.
Ernst, M., & Bülthoff, H. (2004). Merging the senses into
a robust percept. Trends in Cognitive Sciences, 8(4),
162-169.
Forsythe, J., & Kelly, M. (1989). Effects of visual-spatial
added cues on fourth-graders' melodic discrimination.
Journal of Research in Music Education, 37(4), 272.
Frego, D. (1999). Effects of aural and visual conditions on
response to perceived artistic tension in music and
dance. Journal of Research in Music Education, 47(1),
31.
Gembris, H. (2006). The development of musical abilities.
In R. Colwell (Ed.), MENC handbook of musical
cognition and development (pp. 124). New York:
Oxford University Press.
Gordon, E. (1997). Learning sequences in music: Skill,
content, and patterns. Chicago, IL: GIA.
Hidaka, S., Manaka, Y., Teramoto, W., Y, S., Miyauchi,
R., Gyoba, J., et al. (2009). Alternation of sound
location induces visual motion perception of a static
object. PloS One, 4(12).
Kratus, J. (1991). Growing with improvisation. Music
Educators Journal, 78(4), 35-40.
Leman, M. (2007). Embodied music cognition and
mediation technology. London: The MIT Press.
Lipscomb, S. (2005). The perception of audio-visual
composites: Accent structure alignment of simple
stimuli. Selected Reports in Ethnomusicology, 12, 37-
67.
MacDonald, M. (2007). Toward formative assessment:
The use of pedagogical documentation in early
elementary classrooms. Early Childhood Research
Quarterly, 22(2), 232-242.
Massumi, B. (2002). Navigating movements. In M.
Zournazi (Ed.), Hope: New philosophies for change
(pp. 210–244). Annandale, Queensland: Pluto Press.
Naveda, L., & Leman, M. (2009). A Cross-modal
Heuristic for Periodic Pattern Analysis of Samba
Music and Dance. Journal of New Music Research,
38(3), 255-283.
Ng, K., Larkin, O., Koerselman, T., & Ong, B. (2007). i-
Maestro gesture and posture support: 3d motion data
visualisation for music learning and playing. Paper
presented at the EVA 2007 London International
Conference, London.
Ng, K., Weyde, T., Larkin, O., Neubarth, K., Koerselman,
T., & Ong, B. (2007). 3d augmented mirror: a
multimodal interface for string instrument learning
and teaching with gesture support. Paper presented at
the 9th international conference on Multimodal
interfaces, Nagoya, Japan.
Nijs, L. (2008). Evaluatie in het DKO: Het schietlood in
actie. Begeleid zelfstandig leren, Alternatieve
evaluatie 6(21).
Pierce, A. (2007). Deepening musical performance
through movement: the theory and practice of
embodied interpretation. Bloomington and
Indianapolis: Indiana University Press.
Rogers, G. (1991). Effect of color-coded notation on
music achievement of elementary instrumental
students. Journal of Research in Music Education,
39(1), 64.
Schutz, M. (2008). Seeing music? What musicians need to
know about vision. Empirical Musicology Review, 3,
83–108.
Scott, J. (2007). Me? Teach Improvisation to Children?
General Music Today, 20(2), 6-13.
Sloboda, J. (2000). Individual differences in music
performance. Trends in Cognitive Sciences, 4(10),
397-403.
Wanderley, M., Vines, B., Middleton, N., McKay, C., &
Hatch, W. (2005). The Musical Significance of
Clarinetists Ancillary Gestures: An Exploration of the
Field. Journal of New Music Research, 34(1), 97-113.
Welch, G., & Adams, P. (2003). How is music learning
celebrated and developed? Southwell, Notts: British
Educational Research Association.
CSEDU 2010 - 2nd International Conference on Computer Supported Education
336