Songs in Music Education: Design and Early Experimentation of a Web
Tool for the Recognition of Harmonic Changes
Federico Avanzini
1 a
, Adriano Barat
`
e
1 b
, Luca A. Ludovico
1 c
and Marcella Mandanici
2 d
1
LIM – Laboratorio di Informatica Musicale, Dipartimento di Informatica “Giovanni Degli Antoni”,
Universit
`
a degli Studi di Milano, Via Giovanni Celoria 18, Milano, Italy
2
Conservatorio di Musica “Luca Marenzio”, Piazza Arturo Benedetti Michelangeli 1, Brescia, Italy
Keywords:
Online Music Education, Harmonic Rhythm, Free Web Platform, COVID-19.
Abstract:
This paper deals with Harmonic Touch, a Web platform designed to foster the practice of tonal harmony
also in young children. The work focuses on one of the experiences provided by the framework, namely the
gamification of harmonic change recognition in songs. The platform, specifically equipped with new features
to accommodate the needs of teachers during the COVID-19 pandemic, has been tested in two Italian schools
in February 2021. Early experimental results about the main difficulties encountered by the children during
the games are presented and discussed.
1 INTRODUCTION
The Coronavirus disease 2019 (COVID-19) pan-
demic pointed out suddenly and with little room for
doubts the importance of the use of communication
and information technologies in educational contexts.
Teachers of all types and levels have been forced to
learn distance-teaching technologies and change their
working habits overnight, often without any prepara-
tion. As a consequence, learning technologies have
gained much more attention than before, and a grow-
ing number of teachers are eager to discover new
ways to integrate technology into their daily pedagog-
ical practices.
Moreover, the atmosphere of uncertainty that is
characterizing school activities makes the use of on-
line educational platforms extremely useful and func-
tional to various forms of blended learning. As a mat-
ter of fact, web-based tools allow regular school work
to be easily complemented with online activities out-
side the school, and, in case of school closure, they
can fill the gap by ensuring continuity to educational
activities.
In this context, we have released a Web prototype
designed and implemented in a pre-COVID-19 era. It
a
https://orcid.org/0000-0002-1257-5878
b
https://orcid.org/0000-0001-8435-8373
c
https://orcid.org/0000-0002-8251-2231
d
https://orcid.org/0000-0003-1863-4178
adopts the principles of gamification to foster the de-
velopment of a number of tonal-harmony abilities and
competences. The prototype has been tested in two
Italian schools in February 2021, in a period charac-
terized by lessons given partially in presence and par-
tially from remote. Primary-school teachers decided
to focus only on one of the experiences offered by
the platform. In this activity, well-known songs are
employed to make children detect harmonic changes,
i.e. the timed occurrence of new chords. The experi-
mental setup starts from the assumption that popular
music is a vital part of children’ lives. Although the
use of popular music in schools has aroused over the
years a wide debate among educators (Woody, 2007),
the changes in the educational approach, the necessity
of facing more informal education styles and of re-
shaping the contents of music programs are sufficient
reasons to overcome any contraindications (Dunbar-
Hall and Wemyss, 2000; Hebert, 2011). Popular mu-
sic has the power of engaging children in social and
emotional learning, a set of skills that promote chil-
dren’ well-being and enhance cognitive performance
(Campayo–Mu
˜
noz and Cabedo–Mas, 2017). Beyond
these important educational features, songs have a
clear and sound musical structure. Choruses, refrains
and bridges with their standardized functions and du-
rations help the perception of melodic and harmonic
elements, and for this reason are easy to spot and
memorize.
The remainder of the paper is structured as fol-
Avanzini, F., Baratè, A., Ludovico, L. and Mandanici, M.
Songs in Music Education: Design and Early Experimentation of a Web Tool for the Recognition of Harmonic Changes.
DOI: 10.5220/0010540407090720
In Proceedings of the 13th International Conference on Computer Supported Education (CSEDU 2021) - Volume 1, pages 709-720
ISBN: 978-989-758-502-9
Copyright
c
2021 by SCITEPRESS Science and Technology Publications, Lda. All rights reserved
709
lows. Without claiming to be exhaustive, Section 2
mentions other categories of Web tools for music ed-
ucation, and for tonal harmony specifically. Section 3
shortly describes the milestones that brought to the
current release of Harmonic Touch. Section 4 ex-
plains the role played by harmonic rhythm in the per-
ception of music. Section 5 presents the design princi-
ples and the graphical aspects of the Web interface for
the experience under exam. Section 6 describes the
early results collected in the first month of experimen-
tation. As reported in Section 6.3, the platform has
been equipped with new materials and functionalities,
with customized work spaces for teachers and stu-
dents. User performances are shown in Section 6.4.
Section 7 discusses the results collected during the
early experimentation phase. Finally, Section 8 draws
the conclusions and presents the future perspectives
and the expected evolution of the project.
2 RELATED WORKS
The World Wide Web offers many platforms and tools
for music education. Based on their functionalities
and educational objectives, they can be subdivided
into various categories:
1. Online music studios, which provide users with
the functionalities of music sequencers, including
music loops and sounds for song creation (e.g.,
BandLab,
1
Soundtrap,
2
Groovy Music Cloud Edi-
tion, and Soundation
3
);
2. Music performance platforms, which offer some
practical tools to help students learn how to
play an instrument or sight-read music at dis-
tance (e.g., SmartMusic,
4
Doozzoo,
5
Practice-
First,
6
Sight Reading Factory,
7
and Yousician
8
);
3. General music education platforms for devel-
oping listening skills, theory, and composition
(e.g., BrainPOP Arts and Music,
9
PBS KIDS Mu-
sic Games,
10
Classics for Kids,
11
and Focus on
Sound
12
);
1
https://edu.bandlab.com/
2
https://www.soundtrap.com/edu/
3
https://soundation.com/
4
https://www.smartmusic.com/
5
https://doozzoo.com/
6
https://www.musicfirst.com/applications/practicefirst/
7
https://www.sightreadingfactory.com/
8
https://yousician.com/
9
https://www.brainpop.com/artsandmusic/
10
https://pbskids.org/games/music/
11
https://www.classicsforkids.com/
12
https://www.focusonsound.com/
4. Music notation and creation platforms (e.g.,
Flat,
13
Noteflight,
14
and O-Generator
15
);
5. Music theory and ear training platforms (e.g.,
EarBeater,
16
Musictheory.net,
17
and Musition
18
).
Even if the present review is far from being exhaus-
tive, it can provide a broad idea of the educational
offer for music education currently available on the
Web.
The first observation is that nearly all the men-
tioned platforms are not free, and, consequently, they
require the purchase of a premium access to fully ben-
efit of the services offered. If, on one side, paid ser-
vices can guarantee a good quality of contents and
performances, on the other side they do not respond
to the needs of students and teachers who still need to
be encouraged in the use of learning technologies.
As far as tonal harmony is concerned, theoretical
knowledge is usually embedded inside music theory
programs. The typical approach of web platform is to
provide an online version of traditional music theory
books, possibly augmented through the integration of
interactive tools for chord calculation or identifica-
tion. The targeted audience is made of music-theory
students engaged in formal learning programs. For
this reason, such tools can be profitably applied to the
academic study of harmony, but they are not suitable
for children or adults interested in understanding the
phenomenon rather than the theory behind it.
A less formal approach is employed in the Hook-
pad Musical Sketchpad,
19
a software where the
chords of a song are displayed under a piano-roll win-
dow by employing traditional chord notation and dif-
ferent color codes for each chord. The piano-roll in-
terface can be modified by changing pitches, keys,
meter, tempo and chord progressions.
Another interesting tool for the study of harmony
is Mapping Tonal Harmony,
20
a software which dis-
plays in real time the movement of chords in the har-
monic space. It can also be used for very complex
musical pieces both in classic and jazz style, but it is
suitable for professional rather than amateur use.
13
https://flat.io/
14
https://www.noteflight.com/
15
http://www.o-generator.com/
16
https://www.earbeater.com/
17
https://www.musictheory.net/
18
https://www.risingsoftware.com/musition
19
https://www.hooktheory.com/hookpad
20
https://mdecks.com/mapharmony.phtml
CSME 2021 - 2nd International Special Session on Computer Supported Music Education
710
3 BACKGROUND WORK
Under the pressure of the dramatic uncertainty and
discontinuity that is characterizing the life of school
institutions in Italy, we publicly released Harmonic
Touch, a Web platform designed and implemented a
few months before COVID-19 pandemic (Avanzini
et al., 2020). The original idea had been presented and
discussed during a workshop titled “Didattica della
musica e linguaggi digitali” (i.e., music education and
digital languages) in occasion of the 2
nd
edition of
Fiera Didacta Italy, Florence, October 18-20, 2018
(Avanzini et al., 2019a). The audience, mainly com-
posed by primary and lower-secondary school teach-
ers, had the opportunity to test the early prototype and
make remarks that guided a later re-implementation.
Harmonic Touch aims to drive the attention of
teachers towards the possibility of using tonal har-
mony features in music education even from a very
early age. In order to achieve this goal, the plat-
form offers a number of activities following the prin-
ciples of gamification. The user is not pushed towards
gaining theoretical knowledge, as in formal music
education, but rather to develop harmony awareness
through musical perception.
The platform currently presents 3 different expe-
riences:
1. The first experience deals with the perception of
implied harmony, where the user is asked to lis-
ten to a music excerpt and try to pick the implied
harmony - the tonic - from a set of 6 chords repre-
senting the harmonic space. The implied harmony
is the chord that best fits all the melody excerpt;
2. The second experience deals with the detection
of the onsets of the harmonic changes under a
melody while the tune is playing. This kind of ex-
ercise is the focus of the present paper, so it will
be described in detail in Section 5.2;
3. The third experience consists in melody harmo-
nization. After carefully listening to a melody and
exploring the harmonic space, the user is asked
to click on the chord sequence that best fit the
melody. This type of exercise is a combination
of the previous two. In our vision, it represents
the final step on the way towards the development
of tonal harmony awareness.
For this experimental session, we decided to focus our
attention on the second experience, namely the recog-
nition of harmonic changes. Being the most playful
thanks to its gamification features, it has been pos-
itively accepted by school teachers and enthusiasti-
cally welcomed by students.
4 HARMONIC RHYTHM
Harmonic rhythm plays an important role in rhythmic
and melodic perception, as well as in the segmenta-
tion process that governs the way listeners interpret
and group surface musical events (e.g. the notes of a
melody).
Harmonic rhythm has been defined by as the “un-
derlying changes in harmony” (Burkat, 1944) or “the
perception of rhythm that depends on changes in as-
pects of harmony” (Swain, 2002). Although the lat-
ter definition implies that the perception of harmonic
changes may depend on various musical elements,
such as timbre, musical style, the leading of the bass
line, chord density and texture, for the purposes of
the educational activities described in this work we
focus only on two main aspects of harmonic rhythm.
As described by Dawe (Dawe et al., 1993), harmonic
rhythm can be considered in two different ways: as
the sequence of the onsets of different harmonies, or
as a “composite rhythm” that is formed by a sequence
of events belonging to the same chord. This is the case
of the melody of the nursery rhyme “London bridge”
which, despite its simplicity, shows an unexpectedly
complex underlying structure. The arcs in Figure
1 embrace the 2 semiphrases that form the melody;
they are perfectly regular and symmetric. The under-
lying structure of the harmonic rhythm, conversely,
presents different durations: 2 bars tonic (T), 1 bar
dominant (D) and, again, 1 bar tonic (T). The same
structure is repeated in the second semiphrase. How-
ever, as the same chord (T) ends the first semiphrase
and begins the second, the overall duration of the
third chord is 3 bars. Thus, a regular 4 + 4 struc-
ture is superimposed to an irregular harmonic rhythm
(2 + 1 + 3 + 1 + 1). This is due to the effect of com-
posite rhythm (shown in brackets in Figure 1), which
happens whenever the rhythmic accent expected at
the beginning of a new bar is not matched with a
change in the harmony. Examples are provided in
bars 2, 5, and 6 of the excerpt. The regular structure
of the 2 semiphrases makes the rhythmic accent at
bar 5 particularly important but, in spite of this, there
is no change in the harmonic rhythm. This example
demonstrates how the perception of a simple nursery
rhyme can hide a complex harmonic structure, and
what is the role of harmonic rhythm in building such
a richness of musical events.
As theorized by Lerdahl and Jackendoff (Lerdahl
and Jackendoff, 1996), tonal music is built through
the superimposition of 4 main layers, each contain-
ing different musical elements. The most superfi-
cial is the grouping structure, which refers to the
patterns created by the notes of the melody (themes
Songs in Music Education: Design and Early Experimentation of a Web Tool for the Recognition of Harmonic Changes
711
7 8654321
D T
G C(C)(C)
T
C
D
G(C)
T
C
LondonBridge
Figure 1: Harmonic changes of “London bridge”. Slurs embrace 4-bar semiphrases. The lines below chord symbols indicate
the occurrence of chord changes and the extension of the corresponding harmonic areas. Composite rhythm is expressed by
bracketed chord indications at bars 2, 5 and 6.
and phrases). By applying the Gestalt theory prin-
ciples of similarity, proximity and symmetry (Ellis,
1999), listeners are able to partition a melody into
small segments according to the perception of beat
accents and short-term memory (Bigand, 1993). At
lower levels, there are the metrical structure, which
considers the organization of strong and weak beats,
and the time-span reduction and the prolongational
reduction, which represent structural accents of mu-
sical events and tension-relaxation dynamics, respec-
tively (Hansen, 1011).
The process of melody segmentation is not only a
cognitive representation of tonal melody. It has also
been employed as a computational model for the de-
scription of musical activities and for the accomplish-
ment of musical tasks, such as music query and mu-
sic information retrieval (Zhu and Kankanhalli, 2004;
Hirai and Sawada, 2019).
In the case of the second experience of Harmonic
Touch, we provide a segmentation of the melody
based on the occurrence of a new chord instead on the
metrical organization of the melody. As shown by the
case of “London bridge”, this can increase the level
of complexity in the musical perception, forcing the
listener to distinguish between metric and harmonic
accents. This is one of the skills targeted by the sec-
ond experience of Harmonic Touch.
5 THE WEB TOOL
Harmonic Touch is a free and cross-platform tool, cur-
rently supporting Italian and English language. It is
available at the following URL: http://harmonictouch.
lim.di.unimi.it/.
The front-end section and the underlying database
that manages accounts and gathers performance data
have already been designed and released. The back-
end area at the moment of writing is still under devel-
opment, currently supporting basic account options
(such as password change) and, for teacher accounts
only, the configuration of classes and the selection
of exercises. The personal preparation and upload of
new materials through the Web interface, very useful
functions in a crowd-sourcing perspective, have not
been developed yet.
5.1 User Accounts
The landing page lets the user enter his/her creden-
tials. Different user types are supported: an adminis-
tration account reserved for developers, a test account
for authorized beta testers, an anonymous account, a
teacher account, and, finally, a student account. For
the goals of this paper, only the last three types will
be described.
The teacher account grants access to all the experi-
ences and exercises, so that the educator can listen and
test them in order to select the most suitable ones. The
back-office area lets the teacher configure each class
and assign it a specific subset of exercises, also deter-
mining the level of difficulty for the exercise. Even
if a default difficulty level has been assigned by do-
main experts to each piece belonging to the second
experience, such a value is currently under investiga-
tion and should be considered a placeholder; deter-
mining which aspects (e.g., the length of the piece, its
tempo, the regularity in harmonic changes, the type of
chords, etc.) have an impact on the perceived level of
difficulty is one of the goals of the experimentation,
as discussed later.
The student account gives access to the experi-
ences and exercises selected by the teacher for the
class the student belongs to. In other words, a student
can see only a specific subset of the song dataset.
Finally, the anonymous account has been con-
ceived to let a generic user experience the platform in
all its aspects and exercises with no need to be regis-
tered. Performance data from anonymous players are
collected as well, but, in general, they do not allow
to make inferences, since the account is not linked to
a single user, and the aims of accesses can be very
heterogeneous, ranging from curiosity to the explo-
ration of otherwise protected materials. Concerning
the latter aspect, teachers involved in early experi-
mentation discovered that their students had learned
how to overcome the limited piece selection imposed
for their class by using the anonymous account. In
the future, also the anonymous account will present a
small selection of songs for demonstrative purposes.
CSME 2021 - 2nd International Special Session on Computer Supported Music Education
712
5.2 Game Play for Experience No. 2
In order to play Experience No. 2, the user has to com-
plete a 4-step process: i) piece selection, ii) pre-game
listening activity, iii) game play, and iv) game results.
These steps are shown in Figure 2.
After selecting a piece from the list, in the second
step users can listen to it as long as they need in order
to get acquainted with the tune and try to locate chord
changes.
The gaming experience starts in the next step. The
audio track has been segmented into n segments, each
one corresponding to a musical chord; consequently,
Figure 2: From top to bottom, the 4 steps for Experience
No. 2.
harmonic changes are n 1. The interface presents a
treasure map with a dashed path connecting n nodes.
The first node, not numbered, serves as the start but-
ton; the remaining nodes are in correspondence with
timed harmonic changes. When the user clicks over a
node, music is played from that time position.
The ideal behavior, i.e. the one that brings the
player to win with the maximum score, consists in
clicking over nodes step by step, in the right order and
at the right moment (namely when the chord change
occurs). A tolerance of ±0.5 s centered on the cor-
rect timing has been introduced to manage delays and
avoid frustration. Tolerance defines the range within
which user’s click may fall while still being accept-
able.
When trying to detect the harmonic change, the
user has two possibilities of error: anticipated and
delayed clicking. The former case comes from the
perception of a chord change when the tonal area re-
mains unaltered, and it implies a sudden jump to the
new segment of the song (often, the step right ahead
the current one in the dashed path). The latter scenario
refers to the opposite situation, namely the occurrence
of a harmonic change not detected by the user; in this
case, after a short fade-out effect, music stops waiting
for any user action. Please note that, while anticipated
clicking is a type of error that the platform simply
acknowledges (by playing an error sound, forcing a
jump in playback, and decreasing the score), delayed
clicking becomes evident to the user due to the ab-
sence of sound, thus pushing to take an action.
In a gamification approach, score is a typical
means to increase engagement and push players to-
wards better performances. For this reason, each
piece has been associated with a maximum score
s = 100 · (n 1) points, where n is the total number
of tonal areas in the song. A click over the right node
within the ±0.5 tolerance interval adds a score which
is still positive, but reduced on the base of the abso-
lute distance from the right timing: each 0.1 s, the
100-points original amount is cut by 25 points. Click-
ing outside the tolerance window and/or on a wrong
node implies a 10-point penalty. Values of time toler-
ance and score increments/decrements are parametric,
so, in a future evolution of the platform, they could be
customized by the teacher.
The game board presents buttons to confirm re-
sults, restart the match and go back to the pre-game
listening activity. In all these cases, user perfor-
mances are saved in a database where players are
identified by their nickname, class and school only,
so as to protect their privacy; no personal data nor any
other sensitive information are saved in the database.
Songs in Music Education: Design and Early Experimentation of a Web Tool for the Recognition of Harmonic Changes
713
6 EXPERIMENTATION
The experimentation started in two Italian primary
schools in February 2021, in the midst of the resur-
gence of the COVID-19 pandemic. The schools in-
volved were “Istituto Comprensivo Lucilio”, Sessa
Aurunca, Caserta, and “Istituto Ancelle della Carit
`
a”,
Palazzolo sull’Oglio, Brescia. The experimental ac-
tivities started on February 4
th
at “I.C. Lucilio” and
on February 9
th
at “Istituto Ancelle della Carit
`
a”.
Even if educational activities started in presence,
by the end of the month many schools were forced to
switch to remote teaching. This fact necessarily inter-
fered with the experimental plan, which included:
an introductory lesson where teachers had to ex-
plain basic concepts about melody and harmony
and show students how to use the platform;
a pre-test session where the initial level of chil-
dren has been recorded through in-class games
with Songs No. 1, 2, and 3.
a number of lessons where the games on the plat-
form should have been accompanied by explana-
tions by the teachers and face-to face and physical
activities, as described in (Mandanici et al., 2019);
a final post-test, planned at the end of the fourth
week of activity in order to evaluate the advances
obtained.
Due to the forced switch to remote teaching, only a
part of the pre-test phase could be realized in presence
(see Figure 3), while the experimentation – currently
under way at distance will miss all the planned face-
to-face activities.
Anyway, Harmonic Touch is a Web-based frame-
work, consequently it is able to support different use
modes without changing the interaction paradigm,
thus reducing the differences that exist between su-
pervised in-class activities, supervised out-of-school
activities, and unsupervised home activities.
6.1 Aims
For the purposes of the present paper and in the un-
certainty of being able to continue and control the ex-
perimentation in a remote teaching situation, we con-
sider pre-test results (obtained during the first lesson
in each school) and the data gathered during unsuper-
vised experimentation in a 7-day period after the date
of first access.
The analysis concerns three main research ques-
tions:
RQ1 the evaluation of user performances for each
song;
Figure 3: In-class activities during the pre-test phase.
RQ2 the evaluation of user performances for each
harmonic change in relation to the musical char-
acteristics of the song excerpt;
RQ3 the evaluation of the efficacy of the platform in
extending school work at home.
RQ1 focuses on both personal and aggregated results
achieved by players song by song. Even if this ques-
tion apparently addresses how well players have per-
formed, the real goal is understanding what principles
can drive the choice of song excerpts to be employed
both in pre-test and in unsupervised game sessions.
Game-play data, gathered for the first time in a field
experimentation, can help to unveil the musical char-
acteristics that influence the perception of difficulty.
In preparing materials for the second experience, we
aprioristically assumed that the perceived degree of
difficulty could depend on the following parameters:
excerpt duration, number of harmonic changes, aver-
age frequency of changes, harmonic complexity, ir-
regularity of the harmonic rhythm. Answering RQ1
implies a better understanding of such mechanisms.
RQ2 focuses, once again, on both personal and ag-
gregated results achieved by players, but, in this case,
at a finer level of detail: user performances are as-
sessed on single chord changes. Understanding the
origin of poor results requires the analysis of chord
models, the regularity in harmonic changes, and the
extension of tonal areas. Harmonic changes where
children encountered higher difficulties can give some
useful hints to steer the didactic action and to set the
CSME 2021 - 2nd International Special Session on Computer Supported Music Education
714
13
6
9
8
1211
5
7
8
6
4
5
1098
3
7
4
21
6
3
7
2
5
1
43
15
8
1413
8
76
1211
5
1098
76
21
4
76
3
54
543
2
32
1
1
21
342.11
E
220.47
C
(E)
234.00
Fm6
E
414.70
168.00
G7
Aadd9
343.69
434.36
A
170.74
B(E)
(E)
151.84
C
105.21
Fm6
100.58
Am7
Am
408.40
(E)
141.05
B♭m
C♯m7
290.01
196.79
B♭m6
366.00
E(A)
(E)
272.45
G7
174.48
G4
C♯m7
279.54
162.33
Fm6
B4
546.40
(E)E
164.23
C
B4
440.38
217.35
Fm6
2614.36
A
112.55
C6/dim5
191.22
Fm6
E
C
E
336.03
340.44
Dm
286.93359.96
ADm
329.98
B♭
313.34
G
390.79
F
1107.86
Dm
791.00
Dm(G)
699.18
G
701.31
Am
(F)
3350.17
F(Dm)(Dm)(Dm)
1037.94
G
598.35
C
726.10
F
267.76
(A7)A7
356.73
F
303.90
Dm
239.83263.50
B♭G
349.37
FDm
498.62
Dm(G)
549.74
G(F)
753.56
F
554.41
C
496.53
F
(Dm)Dm
606.14
EmC
6.IWanttoBreakFree
5.Comeunfurbotopolino
4.Oceania-HowfarI'llgo(Italianversion)
3.HouseoftheRisingSun
2.Sayonara
1.SomewhereOvertheRainbow
Figure 4: Scores of the songs used in the experimentation. The lines below chord symbols indicate the occurrence of chord
changes and the extension of the corresponding harmonic areas. Composite rhythm is expressed by bracketed chord indica-
tions. Small numbers below indicate the mean of the absolute values of errors (anticipations and delays) expressed in ms. The
first chord has no number associated since it corresponds to the initial play event, so the error is always null.
post-test materials accordingly.
Finally, RQ3 refers to the potential of the platform
to extend the school activities also at home. This is
a relevant goal in a period where in-presence activi-
ties are so discontinuous and uncertain. The platform
can play a unifying role between face-to-face teach-
ing and online activities, thus mitigating the incon-
veniences and the difficulties of the COVID-19 pan-
demic. In this sense, it is fundamental to measure
also the level of engagement, so as to push students
Songs in Music Education: Design and Early Experimentation of a Web Tool for the Recognition of Harmonic Changes
715
towards personal goals and avoid the sense of frustra-
tion due to repeated failures.
6.2 Subjects
The experimentation has involved so far 46 children
(29 females) from two Italian primary schools: 21
students (15 females) from “Istituto Comprensivo Lu-
cilio” and 25 students (14 females) from “Istituto An-
celle della Carit
`
a”. The children of the former school
are attending 2 classes of the 4
th
grade, the students
of the latter are attending 2 classes of the 5
th
grade.
Their age is comprised between 8 and 11 years.
The two student populations are comparable
concerning age, educational goals, previous music
knowledge, etc., so their performances have been
evaluated without considering their different origin.
6.3 Materials
Music excerpts have been chosen by teachers based
on their educational goals, the characteristics of stu-
dents, and considerations related to amusement and
engagement. The songs selected for the pre-test phase
and for the games to be played at home are listed
in Table 1: they cover popular music (Songs No. 1,
3 and 6), nursery rhymes (Songs No. 2 and 5) and
animation-movie soundtracks (Songs No. 4). Even if
children are expected to be already familiar with most
of them, a pre-game listening phase is available for
each song.
21
While Songs No. 1, 4, and 5 show a regular pace
in harmonic changes, Songs No. 2, 3, and 6 are char-
acterized by different types of irregularities (see Fig-
ure 4).
Song No. 2 presents harmonic changes every two
bars and at a slow tempo, but introduces a prolonged
Dm chord across the phrase that begins at bar 9.
Moreover, in the first beat of bar 9 the audio track
presents a strong percussive sound, which makes it
very difficult for children to resist the temptation to
push the button, even if there is no harmonic change.
Song No. 3 is rather regular except for bar 4,
where the harmonic rhythm stops for a whole bar
on the dominant chord (A7). However this structure
does not conflict with the metrical organization of the
melody.
Song No. 6 is characterized by a very unusual
structure, with a very long tonic chord (E) at the be-
ginning, lasting 6 bars. Then a more regular repetition
of 2-bar harmonies (A, E) is followed by bars 11, 12
21
Please note that music pieces can be listened by en-
tering the platform with the anonymous account, choosing
Experience No. 2 and selecting them from the complete list.
and 13 each with one harmonic change. Moreover
Song No. 6 has the slowest pace of chord changes.
6.4 User Performances
The assessment of user performances is a non-trivial
problem. Which aspects should be measured? What
time offset should be tolerated to consider a real-
time user action as correct? Is the set of parame-
ters presented in Section 6.1 suitable to describe the
perceived level of difficulty of a song? These prob-
lems, partially addressed in a pre-experimental phase
(Avanzini et al., 2019b), will be re-evaluated in light
of experimental results.
Detailed data about user performances are avail-
able at http://harmonictouch.lim.di.unimi.it/results/
early/lucilio.html for “Istituto Comprensivo Lu-
cilio”, and at http://harmonictouch.lim.di.unimi.it/
results/early/ancelle.html for “Istituto Ancelle della
Carit
`
a”. In these pages, results are organized user by
user. Each diagram represents a game, each trial is
one of the attempts made to win in a single game.
At the end of such pages, synoptic tables for each
song are reported, with detailed information about
chord changes: the average error, the variance, the
maximum error due to anticipated (highest negative
value) and delayed (highest positive value) clicking,
the mean of absolute errors, the number of positive
errors, the mean of positive errors only, the number of
negative errors, and the mean of negative errors only.
For the sake of brevity, only the most relevant data
will be presented and discussed in the following.
Concerning RQ1, we analyze the data collected in
both schools, paying particular attention to the aver-
age error obtained during school pre-test and home
activities. In order to obtain a single value for each
song, the process consisted in calculating the average
error measured on all chord changes as detected by
the students of both schools (see Figure 5). The un-
derlying idea was to find an aggregate indicator for
the level of difficulty experienced by students: abso-
lute values closer to 0 should indicate exercises where
the identification of chord changes is simpler; never-
theless, it is worth underlining that the reasons can be
heterogeneous.
The piece that reports the highest error is Song No.
2, followed by Song No. 6, while the piece where chil-
dren performed best is Song No. 5. From the analysis
of these results, difficulty does not seem to depend on
the number of harmonic changes (Song No. 2 has the
lowest value) nor on harmonic complexity (Songs No.
2 and 6 employ the smallest number of harmonies),
but, rather, on the average duration of the chords µ
(see Table 1).
CSME 2021 - 2nd International Special Session on Computer Supported Music Education
716
Table 1: The songs employed in the pre-test and for home activities: d is the duration of the song excerpt in seconds, n
c
is the
number of harmonic changes, µ is the average duration of chords in seconds, n
h
is the number of different harmonies, and r is
a Boolean value concerning the regularity of harmonic changes. Other columns are self-explanatory.
phase song no. title year d key n
c
µ n
h
r
pre-test 1 Over the Rainbow 1990 25.4 C 8 2.82 5 yes
pre-test 2 Sayonara 2009 31.0 Dm 6 4.43 3 no
pre-test 3 House of the Rising Sun 1964 27.2 Dm 13 1.94 5 no
home 4 Oceania 2016 23.7 E 8 2.63 4 yes
home 5 Come un furbo topolino 2009 18.2 C 8 1.14 6 yes
home 6 I Want to Break Free 1984 30.5 E 8 5.08 3 no
Figure 5: Mean of the (absolute) errors on chord changes measured song by song. The tallest bar is associated with the hardest
exercise (Song No. 2), the shortest one with the easiest exercise (Song No. 5).
Songs No. 2 and 6 present the largest average du-
rations µ. Surprisingly, Song No. 3, whose number
of harmonic changes is significantly higher than the
other songs, obtains the second best score with an er-
ror of 51.97 ms. Song No. 3 has also an irregularity
in the harmonic rhythm at bar 4, where the A7 chord
stops for 2 beats; but, in this case, there is no conflict
with the metrical organization of the melody. Any-
way, the next change after the irregularity – occurring
at bar 5 obtains the maximum error of the whole
song: 1107.86 ms.
On the other hand Song No. 5 although charac-
terized by a regular harmonic rhythm has the highest
level of harmonic complexity and the smallest aver-
age chord duration (µ =1.14 s). Evidently the regu-
larity combined with the high speed of change plays
a decisive role in the perception harmonic accents.
In order to answer RQ2, we analyze the occur-
rence of the harmonic changes in Songs No. 2 and
6 where children collected the most relevant errors.
For Song No. 2, the highest error is at the 4
th
har-
monic change, equal to 3350.17 ms (see Figure 4).
This is due to the effect of the superposition of har-
monic rhythm (4 bars of Dm from bar 7 to 10) with
the metric accent of bar 9. For Song No. 6, the most
relevant error, equal to 2614.36 ms, occurs at the 1
st
harmonic change. This is due to the confusing effect
of the very long tonic chord (6 bars of E from bar 1 to
6) which makes it hard for children to locate the next
change at bar 7.
RQ3 can be addressed by comparing the number
of games played during the pre-test sessions and the
games played at home in the 7 days after the pre-test.
We have limited our investigation to one week since it
was the shortest interval before the following lesson.
The pre-test is also the lesson when children experi-
enced the first access to the platform, while, from the
afternoon onward, they were free to play whenever
they wanted.
Table 2 shows the statistics of the games played
during the pre-test and at home. Although there is a
remarkable difference between the two schools in the
number of users who autonomously accessed the plat-
form at home, it is interesting to note that the overall
Songs in Music Education: Design and Early Experimentation of a Web Tool for the Recognition of Harmonic Changes
717
percentage of games played at home is not less than
64% of those played in the pre-test. The percentage
of successful games is much better at home than at
school for the students of ‘Istituto Comprensivo Lu-
cilio” (2% at school, 12% at home), while it is nearly
the same for the students of “Istituto Ancelle della
Carit
`
a” (15% at school, 14% at home). These results
seem to indicate that the game has greatly engaged
children, encouraging some of them to play even out-
side school hours. The gamification approach has
pushed them to experiment autonomously, in some
cases obtaining much better performances.
7 DISCUSSION
While experimental activities are still running and in
the uncertainty of its progress, we can only guess
what the trends will be once the experimentation is
complete. However, from these early results, we can
assume that the real difficulty in the detection of har-
monic changes is mainly connected to the average du-
ration of the chords µ, or equivalently to the speed of
changes.
In the choice of the materials for the pre-test we
tried to avoid songs with frequent harmonic changes,
considered too challenging for children. Unexpect-
edly, slow changes seem to be more problematic,
mainly when they conflict with the metrical structure
of the melody, as it happens in Song No. 2.
Also irregularities in the changes and in the har-
monic structure do not seem to have a deep impact
on success rate. Actually, looking at the errors of
Song No. 6, there is a peak in the first change, but off-
sets dramatically decrease as soon as the song takes a
more regular trend. Song No. 3, even if marked with
a (low) level of irregularity, obtains a good score and
performs better than regular songs such as No. 1 and
4.
On the other hand the successes recorded for Song
No. 5 with a speed of harmonic changes approach-
ing the second suggest that the timing is a critical
point for deciding the levels of difficulty for the exer-
cises of this second experience of Harmonic Touch.
Table 2: User performances collected during in-class pre-
test and home activities.
Lucilio Ancelle
pre-test home pre-test home
Users 22 7 19 14
Games 92 59 488 384
Won 2 9 59 55
Lost 90 50 429 329
The trend that emerges is that a problematic is-
sue is the inability to predict the occurrence of har-
monic changes when distant in time. This prolonged
standby seems to leave the listener without reference
points, causing errors as soon as the next harmonic
change occurs. Consequently, most teaching efforts
in the 4 weeks of experimentation before the post-test
should address the sensitization of the various cases
of composite harmonic rhythm, mainly when it con-
trasts with the metrical structure of the song. This
goal can be achieved through user-tailored exercises,
listening activities, and games on songs with char-
acteristics similar to those proposed in the pre-test.
Also physical games with different groups of children
or different parts of the body marking metrical and
harmonic accents could help, but these activities are
clearly hampered by the current pandemic emergency.
Finally, data about the use of the platform are very
encouraging. Soon after enabling user accounts, some
students began to play independently outside of class
hours, as revealed by the timestamps of game ses-
sions recorded in the database. Another encourag-
ing aspect is the request made by children to increase
the song dataset available. These facts bode well for
the continuation of educational activities even at dis-
tance, and they are an indicator of the engagement
level. At the moment we are unable to predict if
the accesses outside school time will increase or de-
crease as the experimentation proceeds, also because
teaching at distance breaks the traditional division be-
tween school hours and extracurricular activities. Ac-
tually, what is emerging is a new learning structure
where face-to-face activities are increasingly medi-
ated by virtual learning environments. This allows not
only the overcoming of the traditional organization of
time and space dedicated to educational activities, but
changes their inner nature, helping to design new or-
ganizational models and educational practices.
8 CONCLUSION AND FURTHER
WORK
In this paper we have presented one of the compo-
nents of the Harmonic Touch platform, whose gen-
eral goal is to raise awareness about tonal harmony
in young students through a gamification approach.
The tool under exam is an experience to help users to
detect harmonic changes in songs, namely the occur-
rence of new chords. The interface has been designed
as a treasure map, with a number of steps (corre-
sponding to new chords) to complete at the right time
(corresponding to harmonic changes in audio) in or-
der to reach the final goal. We have presented and dis-
CSME 2021 - 2nd International Special Session on Computer Supported Music Education
718
cussed the early results gathered with primary school
students aged 8 to 11 and coming from 4 classes of 2
Italian institutes.
Regarding future works, there are multiple di-
rections to take. The first one concerns the exten-
sion of experimental activities to a wider audience
of users, potentially presenting different characteris-
tics in terms of age, school grade, geographic ori-
gin, previous musical experience and skills, and so on.
Other school teachers interested in the project have al-
ready been contacted, and only the uncertainty due to
the COVID-19 pandemic has stopped the launch of a
larger-scale experimentation. Gathering more data is
expected to make more statistically significant trends
emerge.
Other considerations about the suitability of the
platform for distant education are likely to emerge in
the near future. In fact, so far the platform has been
used either in class, under the direct supervision of
teachers, or at home, in a relaxed out-of-school envi-
ronment. A third way is not only possible, but also
probable: a new lockdown for Italian schools could
foster the adoption of the platform for remote and
synchronous school activities. In this case, it will be
interesting to analyze student performances and com-
pare them to the previously mentioned scenarios.
Another direction for future work concerns plat-
form design and implementation. Field trials high-
lighted a number of minor bugs, promptly exploited
by smarter students to artificially improve their re-
sults. In this sense, our beta testers demonstrated to
be very motivated to cheat the system. An example
was the extremely quick repetition of mouse clicks
over the current change in order to get additional score
points, an effect obtained by exploiting the very nar-
row tolerance window introduced to manage clicks
not perfectly timed. This bug was solved as soon as it
was detected by reading data; anyway, it did not inval-
idate performance analysis since we did not base our
considerations on score points, which are presented in
the interface only to improve motivation. Concerning
the latter aspect, in a future release we are planning
to further enhance gamification aspects, e.g. by in-
troducing hall-of-fame and leaderboard functions in
order to increase user’s engagement. One interesting
remark that came from one teacher is that, in the ab-
sence of such features, students started to share their
scores by taking pictures of their game results and
sending them to the class chat.
An aspect to further investigate concerns the evo-
lution of the engagement level when students, on one
side, get accustomed to the game play, and, on the
other side, have to face harder exercises. The data
collected so far do not allow to make inferences about
the long-term trend and understand to what extent stu-
dent engagement is due to the novelty of the platform
rather than the idea itself.
From the educators’ perspective, the availability
of a web tool to support music-education activities
has been appreciated by the teachers who decided to
take part into this initiative. Anyway, being an early
experimentation that involved 4 classes only, we can-
not make general statements about teacher’s accep-
tance and level of satisfaction. This aspect will be
further investigated in future experimentation, that is
planned to involve a much higher number of schools
and classes.
The song dataset, currently constituted by 44
pieces, will be further extended and customized
thanks to the contribution of educators, based on their
educational goals. New pieces can be prepared offline
by teachers, but, at present, they can be added to the
platform only by system administrators. The working
group is also planning to implement back-end tools to
let teachers upload their materials independently.
ACKNOWLEDGEMENTS
We wish to thank Giuseppina Verile and Ivana Lan-
zanova, the principals of the “Istituto Comprensivo
Lucilio”, Sessa Aurunca, Caserta (Italy) and of the
“Istituto Ancelle della Carit
`
a”, Palazzolo sull’Oglio,
Brescia (Italy), respectively. For their active coopera-
tion in the experimental activities, we are also grateful
to Sonia Vezza, teacher, and Maurizio Botti, Maura
Castelnovo, Ottavia Marini and Sara Valente, students
of the Master Degree in Technologies for Music Edu-
cation of the Conservatory “Luca Marenzio”, Brescia
(Italy).
REFERENCES
Avanzini, F., Barat
`
e, A., Ludovico, L. A., and Mandanici,
M. (2019a). A computer-based approach to teach
tonal harmony to young students. In Proceedings of
the 11
th
International Conference on Computer Sup-
ported Education (CSEDU 2019), volume 1, pages
271–279. SCITEPRESS - Science and Technology
Publications, Lda.
Avanzini, F., Barat
`
e, A., Ludovico, L. A., and Mandanici,
M. (2019b). Metrics for the automatic assessment of
music harmony awareness in children. In Proceed-
ings of the 16
th
Sound & Music Computing Confer-
ence (SMC 2019), pages 372–379. SMC.
Avanzini, F., Barat
`
e, A., Ludovico, L. A., and Mandanici,
M. (2020). A web platform to foster and assess tonal
harmony awareness. In Computer Supported Edu-
Songs in Music Education: Design and Early Experimentation of a Web Tool for the Recognition of Harmonic Changes
719
cation. 11
th
International Conference, CSEDU 2019,
volume 1220 of Communications in Computer and In-
formation Science, pages 398–417. Springer Interna-
tional Publishing, Cham.
Bigand, E. (1993). The influence of implicit har-
mony, rhythm and musical training on the abstrac-
tion of “tension-relaxation schemas” in tonal musical
phrases. Contemporary Music Review, 9(1-2):123–
137.
Burkat, L. (1944). Harvard dictionary of music.
Campayo–Mu
˜
noz, E. and Cabedo–Mas, A. (2017). The role
of emotional skills in music education. British Journal
of Music Education, 34(3):243–258.
Dawe, L. A., Platf, J. R., and Racine, R. J. (1993). Har-
monic accents in inference of metrical structure and
perception of rhythm patterns. Perception & psy-
chophysics, 54(6):794–807.
Dunbar-Hall, P. and Wemyss, K. (2000). The effects of the
study of popular music on music education. Interna-
tional Journal of Music Education, 36:23–34.
Ellis, W. D. (1999). A source book of Gestalt psychology,
volume 2. Psychology Press.
Hansen, N. C. (2010/11). The legacy of Lerdahl and Jack-
endoffs ‘A generative theory of tonal music’. Danish
Yearbook of Musicology, 38.
Hebert, D. G. (2011). Originality and institutionalization:
Factors engendering resistance to popular music ped-
agogy in the usa. Music Education Research Interna-
tional, 5:12–21.
Hirai, T. and Sawada, S. (2019). Melody2vec: Distributed
representations of melodic phrases based on melody
segmentation. Journal of Information Processing,
27:278–286.
Lerdahl, F. and Jackendoff, R. S. (1996). A Generative The-
ory of Tonal Music. MIT press.
Mandanici, M., Ludovico, L. A., Avanzini, F., and Barat
`
e,
A. (2019). Learning tonal harmony through bodily
interactions and gamification. In MERYC19. Coun-
terpoints of the senses. Bodily experiences in music
learning, pages 237–246. Ghent University.
Swain, J. P. (2002). Harmonic rhythm: Analysis and inter-
pretation. Oxford University Press.
Woody, R. H. (2007). Popular music in school: Remixing
the issues. Music Educators Journal, 93(4):32–37.
Zhu, Y. and Kankanhalli, M. (2004). Key-based melody
segmentation for popular songs. In Proceedings of the
17
th
International Conference on Pattern Recognition,
2004. ICPR 2004., volume 3, pages 862–865. IEEE.
CSME 2021 - 2nd International Special Session on Computer Supported Music Education
720