Novel 3D Game-like Applications Driven by Body Interactions for

Learning Specific Forms of Intangible Cultural Heritage

E. Yilmaz

, D. Uğurca

, C. Şahin

, F. M. Dagnino

, M. Ott

, F. Pozzi

, K. Dimitropoulos

F. Tsalakanidou

, A. Kitsikidis

, S. K. Al Kork

4,5

, K. Xu

4,5

, B. Denby

4,5

, P. Roussel

4,5

P. Chawah

6,7

L. Buchman

6,7

, M. Adda-Decker

6,7

, S. Dupont

, B. Picart

, J. Tilmanne

, M. Alivizatou

L. Hadjileontiadis

, V. Charisis

, A. Glushkova

, C. Volioti

, A. Manitsaris

, E. Hemery

12,13

F. Moutarde

12,13

and N. Grammalidis

Argedor Information Technologies, Ankara, Turkey

Institute for Educational Technology, National Research Council (ITD-CNR), Palermo, Italy

Information Technologies Institute, Centre for Research and Technology Hellas, Thessaloniki, Greece

Universite Pierre et Marie Curie - Paris 6, Paris, France

University Paris 3 Sorbonne Nouvelle, Paris, France

Centre National de la Recherche Scientifique, Laboratoire de Phonétique et Phonologie UMR 7018, Paris, France

Phonetics and Phonology Laboratory, LPP-CNRS, Paris, France

Universite de Mons, Mons, Belgium

University College London, London, U.K.

Aristotle University of Thessaloniki, Thessaloniki, Greece

MTCG Lab, Dep. Of Ap. Informatics, University of Macedonia, Thessaloniki, Greece

Robotics Lab, Dep. Of Mathématiques et Systèmes, MINES ParisTech-ARMINES, Paris, France

Signal Processing and Machine Learning Lab ESPCI, Paris, France

Keywords: Human Computer Interaction, Interactive Game, 3D Visualization, Sensorimotor Learning, Game based

Learning.

Abstract: The main objective of the EU FP7 ICT i-Treasures project is to build a public and expandable platform to

enable learning and transmission of rare know-how of intangible cultural heritage. A core part of this platform

consists of game-like applications able to support teaching and learning processes in the ICH field. We have

designed and developed four game-like applications (for Human Beat Box singing, Tsamiko dancing, pottery

making and contemporary music composition), each corresponding to one of the ICH use cases of i-Treasures

project. A first preliminary version of these applications is currently available for further validation,

evaluation and demonstration within the project. We have encountered a number of issues, most of which

derive from the peculiarities of the ICH domains addressed by the project, and many have already been

resolved/ The evaluation results are expected to lead to further optimization of these games.

1 INTRODUCTION

Cultural expression is not limited to architecture,

monuments or collections of artefacts, for which

various systems have been developed for capturing,

analysis and visualization (Kyriakaki, 2014),

(Makantasis, 2014) and (Makantasis, 2013).

Intangible Cultural Heritage” (ICH) is defined as a

part of the cultural heritage of societies, groups or

sometimes individuals and it includes practices,

presentations, expressions, knowledge, skills and

related tools to all of these such as equipment and

cultural sites. This intangible heritage passes from

generation to generation and gives people a sense of

identity and continuity; it is the result of the

continuous interaction of communities and groups

with their nature and history and it promotes respect

for cultural diversity and human creativity.

The main objective of the EU FP7 ICT i-Treasures

project (Dimitropoulos, 2013) is to build a public and

expandable platform to enable learning and

transmission of rare know-how of intangible cultural

heritage. The proposed platform combines lots of

different technologies like multisensory technology,

651

Yilmaz E., U

gurca D., ¸Sahin C., Dagnino F., Ott M., Pozzi F., Dimitropoulos K., Tsalakanidou F., Kitsikidis A., Al Kork S., Xu K., Denby B., Roussel

P., Chawah P., Buchman L., Adda-Decker M., Dupont S., Picart B., Tilmanne J., Alivizatou M., Hadjileontiadis L., Charisis V., Glushkova A., Volioti C.,

Manitsaris A., Hemery E., Moutarde F. and Grammalidis N..

Novel 3D Game-like Applications Driven by Body Interactions for Learning Speciﬁc Forms of Intangible Cultural Heritage.

DOI: 10.5220/0005456606510660

In Proceedings of the 10th International Conference on Computer Vision Theory and Applications (MMS-ER3D-2015), pages 651-660

ISBN: 978-989-758-090-1

 2015 SCITEPRESS (Science and Technology Publications, Lda.)

singing voice synthesis and sensorimotor learning to

leave the beaten path in education and ICH

knowledge transmission.

In i-Treasures project, four main ICH use cases

are selected: i) rare traditional songs with the

following sub use cases: a) Byzantine hymns, b)

Corsican “cantu in paghjella”, c) Sardinian “canto a

tenore” and d) beat box; ii) rare dance interactions

with the following sub use cases: a) Calus dance, b)

Tsamiko dance, c) Walloon dance and d)

contemporary dance; iii) traditional craftsmanship

focusing on the art of pottery making, and finally iv)

contemporary music composition based on

Beethoven, Haydn and Mozart musical pieces.

In order to prevent these ICH expressions from

extinction, we aim to provide a tool that will allow

their transmission to new people. Towards this goal,

four novel game-like educational applications are

currently developed, one for each of the four ICH use

cases, to be studies within i-Treasures project: i) rare

traditional songs with a contemporary singing style

namely Human Beat Box (HBB); ii) rare dance

interactions with a popular Greek dance (Tsamiko)

iii) traditional craftsmanship focusing on the art of

pottery making, and finally iv) contemporary music

composition based on Beethoven, Haydn and Mozart

musical pieces. In the current development phase, a

first version of these game-like applications is

available and is planned to be validated and evaluated

for sensorimotor learning using educational scenarios

that have been already been defined within the

project. These applications will be continuously

updated towards fulfilling satisfying the needs of the

project and will also be extended to cover the needs

of other sub-use cases (e.g. Walloon dance, Byzantine

music, etc.).

The rest of this paper is organized in the following

Sections: Section 2 provides an overview of the

related work in similar game-like applications for

learning purposes. Section 3 summarizes the system

architecture and the design/development principles

for these applications and Section 4 presents an

overview of the four ICH games that were developed.

Finally, Section 5 includes discussion in the lessons

learnt, limitations and future work.

2 RELATED WORK IN

GAME-LIKE APPLICATIONS

FOR LEARNING PURPOSES

The adoption of game-like applications in i-Treasures

follows a quite well consolidated trend in the

Technology Enhanced Learning field. The

educational potential of games has been widely

explored and highlighted by researchers within the

wider research area of Game Based Learning (de

Freitas, 2012), (Gee, 2003), (Van Eck, 2006),

(Hainey, 2010). A variety of games originally

developed with entertainment purposes can be used

for educational purposes (Djaouti, 2011); in addition

we recently assisted to the taking off and to the fast

increasing adoption of Serious Games (SGs), those

games that are explicitly designed for educational

purposes Michael, 2006), (Breuer & Bente, 2010),

(Derryberry, 2007).

SGs are more and more employed to sustain

learning and training in a variety of educational fields

(formal and informal education as well as military,

medical training etc.); this is done for a wide range of

target populations, ranging from children to adults

(Charlier, 2012).

The learning potential of Serious Games (SGs)

has been extensively investigated in recent years.

Despite some contrasting voices (Hays, 2005), their

educational effectiveness has been widely recognized

(Facer, 2007), (Mc Farlane, 2002), (Milovanović,

2009). When asserting the effectiveness of game-

based learning, many authors focus on the nature of

interaction with the game environment, citing aspects

like motivation, flow and immersiveness (de Freitas,

2009), (Garris, 2002).

SGs, in fact, proved to support learning in a more

active and engaging way (Gee, 2003) and, from the

pedagogical viewpoint, they offer advanced

interaction such as the possibility of customizing the

learning paths (Bottino, 2009) and of keeping track of

the learners’ behavior and successes/failures and are

more adaptive to meet the specific users’ learning

needs (de Freitas, 2013). Games seem to be able to

support the learning a wide range of skills (spatial

skills, decision-making, problem solving, etc.) (De

Aguilera, 2003) and are also recognized as supporting

the learning of procedures and gestures, reason why

they are widely adopted in professional training

(Martinez-Durà, 2011)

In the field of Cultural Heritage Education ICT

technologies are increasingly being adopted (Ott,

2011), (Gaitatzes, 2001), (Veltman, 2005) and in

particular, Virtual Worlds are often used to broaden

the opportunity to appreciate cultural contents that are

remote in space and or time. Even though they should

be considered very helpful for widening access to

cultural contents, these applications, for example

Virtual Museums, often are not intrinsically engaging

and sometimes fail in supporting active learning, just

giving the opportunity to access information

VISAPP2015-InternationalConferenceonComputerVisionTheoryandApplications

652

(Mortara, 2014).

This is why in the i- Treasures project it was

decided to develop game-like educational

applications (or, in other words, Serious Games,

developed with the specific purpose of sustaining

specific educational interventions)

As to the digital games available in the Cultural

Heritage (CH) area, (Anderson, 2009) and afterwards

(Mortara, 2014) carried out interesting State of the

Art reviews. While the first one is more focused on

technical aspects, the second sketches a panorama of

the actual use of Serious Games in Cultural Heritage

education. According to (Mortara, 2014) in the field

of CH, SGs of different kind are adopted: from trivia,

puzzle and mini-games to mobile applications for

museums or touristic visits, (e.g. Muse-US

, Tidy

City

) to simulations (e.g. the battle of Waterloo

iii

) to

adventures and role playing games (the Priory

Undercroft

, Revolution

As it could be expected, games are more

widespread in the Tangible Cultural Heritage (TCH)

area, where several different examples can be found

(Ott, 2008). Among these, we can mention

ThIATRO, a 3D virtual environment where the player

acts as a museum curator (Froschauer, 2012), or other

digital artifacts such as My Culture Quest

, which

aims at advertising real collections or even the

History of a Place

vii

, which is integral part of a

museum experience at the Archaeological Museum of

Messenia in Greece.

A number of games for smartphones also exist

like Tate Trumps

viii

and YouTell and “G. Averof”

(Cao, 2011) which, for instance, allow museum

visitors to create and share through smart phones their

own media and stories.

Many games also exist in the area of historical

reconstruction for instance the Battle of Thermopylae

(Christopoulos, 2011) or the Playing History

, which

are mainly based on 3D technology so to closely

recreate the environment in which each single event

happened.

A (smaller) number of games have also been

developed in the field of Intangible Cultural Heritage

transmission, where they are also considered very

promising (Mortara, 2014). Some examples are:

• Icura (Froschauer, 2010), a 3D realistic

environment in which the player learns about

Japanese culture and etiquette, which can raise

cultural interest and support a real pre-trip planning.

• Discover Babylon

, Roma Nova

and

Remembering 7th Street

xii

, that are aimed at raising

awareness about ancient Mesopotamia's contribution

to modern culture, ancient Rome and West Oakland

in the time period post-World-War-II.

• Africa Trail

xiii

and Real Lives 2010

xiv

simulate

a 12,000 mile travel by bicycle through Africa or a

different life in any country of the world (e.g. a

peasant farmer in Bangladesh, or a computer operator

in Poland), respectively.

• Papakwaqa (Huang, 2013), a serious game

about the Atayal minority in Taiwan, particularly

focused on intangible cultural assets like tribal

beliefs, customs, and ceremonies.

3 SYSTEM ARCHITECTURE

The general system architecture of the game-like

applications that have been developed for the four

ICH cases is shown in Figure 1. As shown in this

figure, there is bilateral communication between the

game-like applications (i.e., the 3D Visualization

Module for Sensorimotor Learning - 3DVMSL) and

the ICH capture and analysis modules, which are

different for each use case (e.g. Ultrasound and

optical sensors and microphones for Rare singing,

Kinect sensors for Rare Dancing, Kinect and/or Real

Leap Motion

TMxv

sensors for Pottery and

Contemporary Music Composition). This

communication includes: a) data transfer from ICH

capture sensor/s to the games, and b) control of ICH

capture and analysis modules by the 3DVMSL

module. The 3DVMSL module can start/stop sensor

data capture and trigger the required data

capture/analysis functions. 3DVMSL also has

bilateral communication with a) a Content

Management System (“i-Treasures Web Platform”)

that is responsible for different tasks such as user

profile management, ICH expert recording

management, etc. and b) a Learning Management

System, which contains additional educational

courses and material regarding each use case.

Regarding the internal architecture of the

3DVMSL module, the initializer is responsible of

activating sensor setups and starting the game-like

application. This action is followed by login where

the user credentials are taken from the Learning

Management System (LMS). Thereafter, the game

menu is shown and the user can either see the game

tutorial or select one of the activities offered such as

Activity 1, Activity 2,… or Final Challenge. The

“Observe” screen currently displays pre-captured

expert data (corresponding to the selected Activity) to

the user. In the current version, we used expert data

stored at the local game repository. However, in the

final version of the games, this data will be obtained

from the Content Management System (CMS). This

is why the connection between CMS and 3DVMSL is

Novel3DGame-likeApplicationsDrivenbyBodyInteractionsforLearningSpecificFormsofIntangibleCulturalHeritage

653

displayed with dash lines. The “Practice” functions

allow the user to try to replicate the expert moves and

then receive feedback from the system regarding the

evaluation of his/her performance. In order to capture

the performance, a set of ICH capture modules are

used and there is a bilateral communication between

the 3DVMSL and the in order to transfer sensor data

to the game.

Figure 1: Detailed System Architecture of Game-like

Applications.

All four ICH games are incorporated in a single

game-like application to ensure simplicity and

integrity. Certain game characteristics, e.g. the splash

screen, the authentication and game selection screen,

the observe and practice phases are common in all

ICH games. We used Unity 3D game engine to

develop the games, due to the following reasons: a)

Unity 3D is an industry proven game development

environment, b) It is easy to deploy the games to

various platforms ranging from Desktop PCs to

mobile devices, c) The development team is highly

experienced in Unity. We used 3DMax Studio and

Maya to model the characters and game

environments, namely 3D assets. Those 3D

modelling tools are also very common in game

development industry. We used Photoshop to prepare

2D assets of the games. All of the codes are written

by using C# programming language.

4 GAME-LIKE APPLICATIONS

FOR LEARNING FORMS OF

ICH

4.1 Rare Traditional Singing: The

Human Beatbox (HBB) Game

The Human Beatbox (HBB) is an artistic form of

human sound production in which the vocal organs

are used to imitate percussion instruments, but also

wind and string instruments. Therefore, facial and

intra-oral movements of the tongue, lips and jaw are

involved in sound production.

The game scenario for Human Beatbox (prepared

by the sub-use case leader CNRS together with beat

box expert) is as follows:

The learner will have at its disposal a portable

helmet based system (Jaumard-Hakoun, 2013),

(Chawah, 2014), (Al Kork, 2014). The latter provides

vocal tract sensing techniques developed for speech

production and recognition. It consists of a

lightweight “hyper-helmet” containing an ultrasonic

(US) transducer to capture tongue movement, a video

camera for the lips, and a microphone.

The learner will be guided by some description of

the sound to be produced and will be listening to the

sounds to imitate them. These descriptions can be

related to natural sounds such as imitating animal

sounds or mouth noise.

The learner will also look at the mouth and tongue

of the expert to try to produce different sounds. The

rhythm is an important dimension in HBB and hand

movement or a metronome can be added in the

scenario in the updated future version.

The HBB game will consist initially of a single

lesson, where each isolated sound will be described

and will be demonstrated by the expert (“Observe

mode”) and then imitated by the learner(“Practice

mode”). Later on, the learner will go to the next step

that will be to produce two sounds, then three in a

row.

The learner should practice each activity a number

of times, and finally perform all the activities at once.

The practice screen of the HBB game is illustrated in

Figure 2.

Figure 2: Practice Screen HBB Game.

4.2 Rare Dancing: The Tsamiko Game

Four different sub use cases are examined in the

project: (a) Calus dance (Romania), (b) Tsamiko

dance (Greece), (c) Walloon dance (Belgium) and (d)

contemporary dance. In this first version of the

Activities

#1, #2, …,

Final

Challen

ICH Capture

modules

3DVMSL

VISAPP2015-InternationalConferenceonComputerVisionTheoryandApplications

654

application, a game for Tsamiko dance learning was

implemented. Tsamiko is a popular traditional folk

dance of Greece, done to music of ¾ meter. The game

focuses on teaching the basic steps of Tsamiko to

learners with minimal or no prior dancing experience.

The game scenario was prepared in close cooperation

with Tsamiko dance experts in Thessaloniki and was

the basis for designing the game.

In order to simplify the learning process of

Tsamiko dance, the experts have defined two

activities comprised of several exercises. The first

activity focuses on the simple single step Tsamiko

style (10 steps) and includes five exercises while the

second activity focuses on the more advanced double

step style (16 steps) and includes four exercises.

The learner has to repeat all of the exercises to

complete the activity objective. Each exercise

consists of several dance steps, which are presented

to the learner one by one. In order to proceed to the

next exercise, the learner must repeat the current

exercise at least 3 to 5 times correctly. Some exercises

of the second activity are more challenging and must

be repeated at least 8 times.

At the beginning of each exercise, a video with

the 3D expert virtual avatar performing the specific

moves is shown to the learner. Afterwards, the learner

is expected to try to imitate the same moves correctly.

The system starts the data capture and gets fused

animation data from an Body motion capture module

that acquires animation data via multiple Kinect v1

sensors, fuses the information properly and provides

to the game a single sequence of animation data

corresponding to the user body movements.

If the imitation is completed properly, the learner

can proceed to the next exercise. Otherwise, the

learner is expected to repeat the same exercise till s/he

does the moves correctly. Although the exercises

progress sequentially, in some cases the learner has to

Figure 3: Practice Screen: Backward View of 3D Expert

Avatar (Big Central Window), Animation Controller

(Bottom Center), Exercise Indicator (Top Center), Expert

Video (Top Right), 3D Learner Avatar (Center Right),

Backward View of Close-up Expert Legs (Bottom Right).

repeat not only the previous exercise but also some

more. A screenshot from the Practice Screen is shown

in Figure 3.

One of the most important aspects of the game is

to evaluate the performance of the learner. In order to

achieve this challenging task, moves of the learner

avatar and expert avatar are compared using a Fuzzy

Inference based algorithm (Kitsikidis, 2014). This

approach uses fuzzy inference to not just compare

absolute joint positions, but to evaluate more

meaningful features.

4.3 Traditional Craftsmanship

The traditional craftsmanship use case focuses on

pottery, which is an art of crafting ceramic. The main

objective of this game is to let the learner observe and

practice the basic moves of wheel-throwing

earthenware pottery by using various sensors and

gaming interface.

The following game scenario, prepared with the

close cooperation of pottery experts in Thessaloniki,

is taken as a basis while designing the traditional

craftsmanship application. There are four activities,

which are different from each other and have different

levels of complexity. The expert demonstrates the

pottery making procedure and it is expected from the

learner to imitate the expert to form an object (Figure

4). The first activity focuses on throwing and

centering the clay on the wheel, the second on how to

make the bottom of the object, the third activity

shows how to shape formation with a tool and the

forth how to cut and remove the final object from the

wheel. In the “Final challenge” activity, the learner

tries to perform all activities in a row in order to make

an object. There is a performance evaluation

threshold value for each activity that allows the user

to proceed to the next activity.

The application communicates with the Body and

Gesture Data Capture and Analysis Module that is

Figure 4: Practice Screen: Angled View of 3D Learner

Avatar (Big Central Window), Animation Controller

(Bottom Center), Expert Video (Top Right), Close-up on

Expert Hands (Center Right).

Novel3DGame-likeApplicationsDrivenbyBodyInteractionsforLearningSpecificFormsofIntangibleCulturalHeritage

655

responsible for capturing the hand and upper body

motions of an expert potter. Specifically, the module

can read, process and capture hands animation data

from a single Leap Motion sensor (multiple Leap

Motion sensors will be supported in the near future)

and combine this information with body movement

from multiple Kinect sensors. In addition,

synchronization and fusion of hand and body

animation streams are performed in order to output a

single animation stream containing skeletal

information of both hands and the body of a person.

Also, there is an optional mode available for

accurately tracking wrist positions in space, based on

blob tracking. This mode significantly improves the

quality of captured data but requires the user to wear

round coloured markers around the wrists.

4.4 Contemporary Music Composition

The Contemporary music composition use case aims

to develop a novel Intangible Musical Instrument

(IMI), which is supposed to map natural gestures

performed in a real-world environment to

music/voice segments. Besides, the emotional status

of the performer is also planned to be part of this

multimodal human interface. Thus, the game-like

application is designed in a way to map not only

gestures but also the emotions of the learner.

The Intangible Musical Instrument aims at

capturing piano-like gestures, which means that the

gestures of the performer resemble piano player

gestures. The new gestures are inspired by piano-like

performances and are transformed into sounds via a

“mapping” phase. Another objective, made possible

with the technology used here, is to create a powerful

pedagogical tool. The player then becomes a learner

and interacts with the system in order to master piano-

like techniques.

The current IMI sensor setup (Figure 5) includes:

a) a “stand” made of wood and Plexiglas under

which a Leap motion sensor is placed. b) Two

inertial sensors for wrists (Animazoo motion capture

suit) and c) an Emotiv sensor for capturing emotions.

This prototype setup is integrated into a game

framework where the user/learner is assessed by the

Figure 5: Detailed description of IMI sensor setup.

system. There are two phases: the first one is the

observing phase, in which the learner observes the

expert’s gesture and listens to the corresponding

sound. The second phase is the practice phase, in

which the learner experiments with the expert’s

gestures. The practice phase of gestures involved in

the contemporary music case could be depicted as

follows (Figure 6): a simple table stands in the middle

of a room. This room looks like a recording studio

with several musical instruments dispatched around.

The user can also see the avatar of the learner/expert

player inside the 3D environment and s/he is able to

manipulate both the studio environment and the

avatar so as to get an “objective view” where s/he sees

his/her own hands plus additional views (perspective,

top view). One main idea conveyed in this first

prototype is the use of the table as a reference plane.

The main approach concerning the learning phase

is “learn by doing”. The learner should basically

attempt to perform the gestures introduced by a

virtual expert on a (real) table. The expert gestures are

driven offline from previously recorded gesture data.

In this first prototype, only the effective gestures were

considered (gestures for Ascending/descending

scales and ascending/ descending arpeggios).

The musical game activities include observe and

practice phases. In the observe phase, the learner can

observe the video of an expert performing a musical

gesture.

Figure 6: Practice Screen: Isometric View of 3D Learner

Avatar (Big Central Window), Animation Controller

(Bottom Center), Expert Video (Top Right), Close-up

Expert Hands (Center Right).

4.4.1 EmoActivity Scenario

A special activity (EmoActivity) of IMI is basically

designed to prompt the user to reach and sustain

certain affective states via a gamified process

involving affective images. The activity is also based

on the characterization of affective states by the

valence-arousal model of affect (Russel, 1980).

Valence denotes whether an emotion is positive or

VISAPP2015-InternationalConferenceonComputerVisionTheoryandApplications

656

negative, while arousal constitutes a measure of the

excitation that accompanies the emotion. Thus, each

affective state can be modeled as a point on the plane

defined by the orthogonal axes of valence and

arousal. The EmoActivity will be part of the novel

intangible musical instrument where the user's

affective state will influence the produced musical

content.

The game consists of three levels with each level

having three difficulty levels. In Level 1 the user is

asked to reach a certain affective state and sustain it,

while in Levels 2 and 3 the user is asked to reach two

and three consecutive states, respectively. The

difficulty in each level (easy, medium, hard) lies in

the target affective states that the user is asked to

reach based on his/her current emotional status, e.g.,

if difficulty level is set to easy and the user is calm,

he/she will be asked to reach a target affective state

of excitement as this transition is more easy than a

state of sadness or anger/fear.

5 DISCUSSION: LESSONS

LEARNT, LIMITATIONS,

FUTURE WORK

Four novel 3D game-like applications driven by body

interactions have been developed of four ICH types

within i-Treasures project. In this first version, we

have encountered a number of problems, most of

which derive from the peculiarities of the ICH

domains addressed by the project. Each ICH has its

own specificities and knowledge domains that need to

be made explicit and formalized, but also there are no

consolidated teaching/learning practices, as the

transmission of these cultural expressions relies on

informal situations based on imitation. Thus, the task

of designing and developing games able to support

teaching and learning processes in the ICH field, is

thus per se a huge challenge.

Starting from the definition of the learning

objectives and the contents to be included in the

games, down to the definition of the game dynamics

and the interface, the whole process was the result of

a complex interaction among many variables, some

of them predictable, others completely unforeseen.

More specifically, a first issue to discuss is the

multi-disciplinary structure of the development

(project) team. The i-Treasures consortium is

comprised of partners with different backgrounds

ranging from commercial video game development to

audio research. Thus, this first version of

visualization module harmonizes not only technical

research and 3D game development, but also

educational aspects. Probably blending technical

research and educational targets under a game design

pipeline at the same time was the toughest challenge,

since the expectations differ significantly. This has

led to accept some compromise, especially from the

educational’ and game dynamics’ point of view.

On the other hand, when we look at it from the

technical perspective, communication of sensors and

the visualization module and conversion of the sensor

output into 3D animation were very challenging

issues. However, any of these technical issues have

no meaning to the end users of the i-Treasures project,

unless it is presented in a coherent way in respect to

the learning objectives and educational design.

Therefore, significant amount of time and resources

have been assigned to blending technical work and

educational aspects and parallelization of

development efforts. At the end of this first version,

the development team has reached a common

understanding and raised the required know-how,

thus facilitating future work.

Regarding to the communication between the

sensors and the visualization module, we encountered

many issues, such as different OSs, different types of

same OS (such as Windows 32 bit, Windows 64 bit)

due to variety of sensors, 3

party controllers,

limitations of research tools etc. At the beginning, we

have tried to define a single data transfer framework

based on TCP. However, at later development phases

many modifications have been made and even some

data transfer is shifted from TCP to UDP considering

the data size, data transfer frequency and even the

limitations of the 3

party controllers. Besides, web

services have also been assigned for low-frequency

communications between remote systems. In order to

minimize the workload, we transferred fused data

when we used different sensors. Besides, we defined

a common structure to visualize both the offline and

online version with same software functions. Thus,

the same data template is used to keep expert data and

to transfer learner data in real-time. Even though we

spent additional resources, this approach will save

additional development time for the future versions.

One last issue regarding the communication

between sensors and visualization module is the

bilateral communication. In typical games, game

controllers and sensors are all controlled by the game

in order to ensure full control. However, in this

research project we have to use more experimental

setups, which have no built-in functionality to be

controlled by a game. Thus, the game must be ensure

that everything is OK and calibrated. Besides, the

game should trigger many events, such as starting a

Novel3DGame-likeApplicationsDrivenbyBodyInteractionsforLearningSpecificFormsofIntangibleCulturalHeritage

657

record, data transfer or even request an evaluation

result. For this first version, this issue was quite

challenging. Even though we achieved some sort of

control, there are still significant issues to be solved

for the next versions.

Another lesson learnt is to have more flexibility

for modifications of the games. Although an initial

design was available, when it came to the realization,

there were many modification requests. In a typical

commercial game or educational application most of

the requirements are determined earlier and the

design is realized by following the initial plan.

However, in this work, there are external participants,

such as domain experts, who have no expertise in

software development and games. Thus, we followed

a more iterative approach to get their feedback during

intermediate development steps and update the design

to reflect their contributions and feedback.

Considering the rich variety of ICH content, the

games started to show more diversity. Many things

have been changed, ranging from activity captions to

practice modes. In order to overcome this issue, the

only way is to design an external entity, which is

capable of changing almost every visual aspect of the

game. Therefore, we designed a configuration XML

that can help us define almost everything about the

games. At the beginning, this took significant time

and caused some delays in the planned work.

However, in the next versions and development

cycles this XML will be a life-saving jacket for us,

allowing everything will be fully configurable

without compiling the games.

In conclusion, many of the issues have been

resolved and four solid game-like applications have

been, able to be integrated in the i-Treasures platform

and further updated and extended in the future.

Besides, the latter outputs will be more robust, since

most of the resources will be used for research and

development rather than trying to solve the technical

problems.

ACKNOWLEDGEMENTS

This work is funded by the European Commission via

the i-Treasures project (Intangible Treasures -

Capturing the Intangible Cultural Heritage and

Learning the Rare Know-How of Living Human

Treasures FP7-ICT-2011-9-600676-i-Treasures). It is

an Integrated Project (IP) of the European Union's 7th

Framework Programme 'ICT for Access to Cultural

Resources.

REFERENCES

Dimitropoulos K., Manitsaris S., Tsalakanidou F.,

Nikolopoulos S., Denby B., Al Kork S., Crevier-

Buchman L., Pillot-Loiseau C., Dupont S., Tilmanne J.,

Ott M., Alivizatou M., Yilmaz E., Hadjileontiadis L.,

Charisis V., Deroo O., Manitsaris A., Kompatsiaris I.,

and Grammalidis N., 2014. Capturing the Intangible:

An Introduction to the i-Treasures Project, in Proc. 9th

International Conference on Computer Vision Theory

and Applications (VISAPP2014), Lisbon, Portugal, 5-8

January.

G. Kyriakaki, A. Doulamis, N. Doulamis, M. Ioannides, K.

Makantasis, E. Protopapadakis, A. Hadjiprocopis et al.,

2014. 4D Reconstruction of Tangible Cultural Heritage

Objects from Web-Retrieved Images, in International

Journal of Heritage in the Digital Era 3, no. 2 (2014):

431-452.

K. Makantasis, A. Doulamis, N. Doulamis, M. Ioannides,

N. Matsatsinis, 2014. Content-Based Filtering for Fast

3D Reconstruction from Unstructured Web-Based

Image Data, Int. Conf. on Cultural Heritage (Euromed

2014), Lemessos, Cyprus, 3-8 November.

K. Makantasis, A. Doulamis, N. Doulamis, 2013. A non-

parametric unsupervised approach for content based

image retrieval and clustering. Proc. of 4th ACM/IEEE

international workshop on Analysis and retrieval of

tracked events and motion in imagery stream, ACM.

de Freitas, S., Earp, J., Ott, M., Kiili, K., Ney, M., Popescu,

M., Stanescu, I., 2012. Hot Issues in Game Enhanced

Learning: the GEL Viewpoint. Procedia Computer

Science, 15, 25-31.

Gee, J.P., 2003. What Video Games Have to Teach Us about

Learning and Literacy. NY: Macmillan.

Van Eck R, 2006. Digital Game-Based Learning: It's Not

Just the Digital Natives Who Are Restless.

EDUCAUSE Review, 41( 2), 17-30.

Hainey, T., 2010. “Using Games-Based Learning to Teach

Requirements Collection and Analysis at Tertiary

Education Level”, Thesis submitted in partial fulfilment

of the requirements of the University of the West of

Scotland for the award of Doctor of Philosophy.

Djaouti D., Alvarez J., Jessel J.-P., 2011. Classifying

Serious Games: The G/P/S Model, Patrick Felicia (ed),

"Handbook of Research on Improving Learning and

Motivation through Educational Games: Multidisciplinary

Approaches", (pp.118-136), IGI Global.

Michael, D., Chen, S., 2006. Serious games: Games that

educate, train, and inform. Boston, MA.: Thomson

Course Technology.

Breuer J., Bente G., 2010. Why so serious? On the Relation

of Serious Games and Learning - Eludamos. Journal for

Computer Game Culture, 4 (1) 7-24.

Derryberry A., 2007. Serious games: online games for

learning. (online), http://www.adobe.com/ resources/

elearning/pdfs/serious_games_wp.pdf.

Charlier, N., Ott, M., Remmele, B., Whitton, N., 2012. Not

Just for Children: Game-Based Learning for Older

Adults. In 6th European Conference on Games Based

Learning, Cork, Ireland (pp. 102-108).

VISAPP2015-InternationalConferenceonComputerVisionTheoryandApplications

658

Hays, R.T., 2005. The effectiveness of instructional games:

a literature review and discussion, Naval Air Warfare

Center Training Systems Division,

http://adlcommunity.net/file.php/ 23/GrooveFiles/

Instr_Game_Review_Tr_2005.pdf.

Facer, K., Ulicsak, M., Sandford, R., 2007. Can Computer

Games Go to School …?, BECTA http://partners.becta.

org.uk/index.php?section=rh&rid=13768.

Mc Farlane, A., Sparrowhawk A., Heald, Y., 2002. Report

on the educational use of games: an exploration by

TEEM of the contribution which games can make to the

education process. Teem, Cambridge.

Milovanović, M., Minović, M., Kovačević, I., Minović, J.,

Starčević, D., 2009. Effectiveness of Game-Based

Learning: Influence of Cognitive Style. In: Lytras,

M.D., Ordonez de Pablos, P., Damiani, E. Avison, D.,

Naeve, A., Horner, D. G. (eds.) Best Practices for the

Knowledge Society. Knowledge, Learning,

Development and Technology for All. Communications

in Computer and Information 49 (pp. 87-96). Science

Springer, Heidelberg.

de Freitas, S., Neumann, T., 2009. The use of ‘exploratory

learning’ for supporting immersive learning in virtual

environments. Computers and Education, 52(2): 343-352.

Garris, R., Ahlers, R., Driskell, J. E., 2002. Games,

motivation, and learning: a research and practice model.

Simulation & Gaming, 33(4), 441–467.

Bottino, R. M., Ott, M., & Benigno, V. (2009). Digital mind

games: experience-based reflections on design and

interface features supporting the development of

reasoning skills. In Proc. of 3rd European Conference

on Game Based Learning (pp. 53-61).

de Freitas, S., Ott, M., Popescu, M. M., Stanescu, I., 2013.

Game-Enhanced- Learning: Preliminary Thoughts on

curriculum integration. New Pedagogical Approaches

in Game Enhanced Learning: Curriculum Integration.

IGI Global.

De Aguilera, M., Mendiz, A., 2003. Video games and

education: (Education in the face of a "parallel school").

ACM Computers in Entertainment, 1(1), 10-10.

Martínez-Durá, M., Arevalillo-Herráez, M.,García-

Fernández, I., Gamón-Giménez, M.A., Rodríguez-

Cerro, A. 2001. Serious Games for Health and Safety

Training. In M.Prensky, (Ed). Digital game-based

learning. New York: McGraw-Hill.

Ott, M., Pozzi, F., 2011. Towards a new era for Cultural

Heritage Education: Discussing the role of ICT.

Computers in Human Behavior, 27(4), 1365-1371.

Gaitatzes A., Christopoulos, D., Roussou, M., 2001.

Reviving the past: cultural heritage meets virtual

reality. In Proc. of Virtual reality, archeology, and

cultural heritage, pp. 103-110.

Veltman K.H., 2005. Challenges for ICT/UCT Applications

in Cultural Heritage. In: Carreras, C. (ed.) ICT and

Heritage, online at http://www.uoc.edu/digithum/

7/dt/eng/dossier.pdf.

Mortara, M., Catalano, C.E., Bellotti, F., Fiucci, G.,

Houry-Panchetti, M., Petridis P., 2014. Learning

Cultural Heritage by serious games. Journal of Cultural

Heritage, 15(3), 318–325.

Anderson, E. F. McLoughlin L., Liarokapis F., Peters, C.,

Petridis, P., de Freitas, S., 2009. Serious Games in

Cultural Heritage. M. Ashley and F. Liarokapis

(Editors). In Proc. of the The 10th International

Symposium on Virtual Reality, Archaeology and

Cultural Heritage VAST - State of the Art Reports.

Ott, M., & Pozzi, F., 2008. ICT and Cultural Heritage

Education: Which Added Value?. In Lytras et al (Eds).

Emerging Technologies and Information Systems for the

Knowledge Society. Lecture Notes in Computer Science,

5288 (pp.131-138). Springer Berlin Heidelberg.

Froschauer, J.; Arends, M.; Goldfarb, D., Merkl, D., 2012.

A serious heritage game for art history: Design and

evaluation of ThIATRO. In 'VSMM', 18th International

Conference in Virtual System and

MultimediaConference, IEEE, pp. 283-290.

Cao, Y., Klamma, R., Jarke, M., 2011. The Hero's Journey

– template-based storytelling for ubiquitous multimedia

management. Journal Multimedia, 6 (2) 156–169.

Christopoulos, D., Mavridis, P., Andreadis, A.,

Karigiannis, J.N., 2011. Using virtual environments to

tell the story: The battle of Thermopylae. In VS-Games

2011, pp. 84-91.

Froschauer, J., Seidel, I., Gartner, M., Berger, H., Merkl D.,

2010. Design and evaluation of a serious game for

immersive cultural training. In 'VSMM', 16th

International Conference in Virtual System and

MultimediaConference, IEEE, pp. 253–260.

Huang, C., Huang, Y., 2013. Annales school-based serious

game creation framework for Taiwan indigenous

cultural heritage. Journal of Computing in Cultural

Heritage, 6 (2).

Jaumard-Hakoun A., Al Kork, S.K., Adda-Decker M.,

Amelot A., Buchman L., Fux T., Pillot-Loiseau C.,

Stone M., Dreyfus G and Denby B., 2013. «Capturing,

Analyzing, and Transmitting Intangible Cultural

Heritage with the i-Treasures Project» in Ultrafest VI.,

Edinburgh, UK.

Chawah P., Al Kork S. K., Fux T., Adda-Decker M.,

Amelot A., Audibert N., Denby B., Dreyfus G.,

Jaumard-Hakoun A., Pillot-Loiseau C., Stone M., Xu

K. and Buchman L., 2014. «An educational platform to

capture, visualize and analyze rare singing,» in

Interspeech Conference, Singapore.

Al Kork S. K., Jaumard-Hakoun A., Adda-Decker M.,

Amelot A., Buchman L., Chawah P., Dreyfus G., Fux

T., Pillot C., Roussel P., Stone M., Xu K. και Denby B.,

2014. A Multi-Sensor Helmet to Capture Rare Singing,

An Intangible Cultural Heritage Study, 10th

International Seminar on Speech Production, Cologne,

Germany.

Kitsikidis A., Dimitropoulos K., Yilmaz E., Douka S.,

Grammalidis N., 2014. Multi-sensor technology and

fuzzy logic for dancer’s motion analysis and

performance evaluation within a 3D virtual

environment, HCI International 2014, Heraklion,

Greece, 22-27 June.

Russell, J. A., 1980. “A circumflex model of affect,”

Journal of Personality and Social Psychology, vol. 39,

1161-1178.

Novel3DGame-likeApplicationsDrivenbyBodyInteractionsforLearningSpecificFormsofIntangibleCulturalHeritage

659

Coenen T., 2013. MuseUs: case study of a pervasive cultural

heritage serious game. Journal on Computing and Cultural

Heritage (JOCCH), 6(2), 8:2-8:19

http://totem.fit.fraunhofer.de/tidycity

iii

http://www.bbc.co.uk//history/british/empire

seapower/launch gms battle waterloo.shtml

A. Doulamis, F. Liarokapis, P. Petridis, G. Miaoulis. 2011.

Serious games for cultural applications. In D. Plemenos, G.

Miaoulis (Eds.), Artificial Intelligence Techniques for

Computer Graphics, Springer. The game is a reconstruction

of the Benedictine monastery in Coventry, dissolved by

Henry VIII

Francis R., 2006. Revolution, learning about history through

situated role play in a virtual environment. Proceedings of

the American educational research association conference.

The game is a role-playing game in the town of colonial

Williamsburg during the American Revolution.

http://www.mylearning.org/my-culture-

quest/interactive/587/

vii

http://www.makebelieve.gr/mb/www/en/portfolio/museums

-culture/54-amm.html

viii

http://www.hideandseek.net/tate-trumps/

www.playinghistory.eu

http://www.fas.org/babylon/

http://www.seriousgamesinstitute.co.uk/research. aspx?item

=171§ion=14

xii

http://7thstreet.org/

xiii

http://www.mobygames.com/game/africa-trail

xiv

http://www.educationalsimulations.com/products.html

https://www.leapmotion.com/

VISAPP2015-InternationalConferenceonComputerVisionTheoryandApplications

660