Explore Through the Past: Gesture-Based Mobile Game for Children

Observing Geological Layer Exhibit at History Museum

Kihiro Tokuno

, Fusako Kusunoki

and Shigenori Inagaki

Information Design Department, Tama Art University, Hachioji, Toyko, Japan

Department of Education, Rikkyo University, Toyoshima, Tokyo, Japan

Keywords: Gamification, Gesture, Geo-Science Education, Interaction Design.

Abstract: Geo-Science education enables children to enjoy observing and investigating environment around them. In

particular, physically experiencing geological formations in a museum or other settings can enhance the

effectiveness of geo-science education. Mobile devices are widely spread and have been used in various areas.

However, during their usage, the physical movements of learners are restricted. Restriction in physical

movements leads to reduction in enjoyment. Therefore, our study developed an action game wherein learners

interacted with a digital object on a mobile device through a character controlled by their body movements.

Our experiment result shows children found the game engaging and educational and were willing to try again.

The game was easy to use and natural, suggesting it increased interest in the exhibit. The results suggest the

use of gesture-based learning experiences can increase interest and make learning about geological formations

in museums more effective.

1 INTRODUCTION

1.1 Introduction

Geo-Science education makes children more

interested in the environment around them and

enables them to enjoy observing and investigating

things that interest them. It provides a variety of

knowledge about the environment, such as past

crustal movements and environmental changes, the

organisms that inhabited the environment, and how

our ancestors lived in the past. As a first opportunity

to learn geology, observational studies are required in

the elementary school curriculum. However, it is

difficult for children to gain an essential

understanding of geology through classroom learning

alone (Ford, 2005). Although students visiting

outcrops directly for observation and learning is

desirable, in this highly urbanized environment,

children can visit only a limited number of outcrops

easily and safely. Meanwhile, Geoparks are

becoming increasingly popular around the world as

places where geology can be studied (Arima, 2016).

One problem with geoparks, however, is that in

addition to safety, the learning experience is largely

dependent on the prior knowledge of the individual.

The museum is much more convenient, safe, and has

stable resources for geological learning. A stripped

section of a geological stratum installed as an exhibit

can provide an experience similar to actual

observation and learning. However, most museum

exhibits are valuable academic materials, which are

often fragile and lack durability. Therefore, many

museums prohibit visitors to touch the exhibits, only

allowing them to observe. Hence, exhibitors must

read the exhibit panels and recognize the labels

attached to understand the contents. However, many

geological contents are inherently unobservable

because of the scale of time and space, making

observations extremely complex and difficult for

children to understand (Trend, 1998). It is difficult to

be interested in an exhibit based on its appearance

alone, and the background and history of an exhibit

are important too (Gerven et al., 2018). Consequently,

geological exhibits are difficult to captivate

children’s interest.

For intrinsic motivation through an even more

intense learning experience, learning while having

fun is effective (Henderson et al., 2007). Learning

games are effective for such learning experiences.

Children have been found to believe that problem

solving and exploration in games helped them learn

(El Mawas et al., 2019). Gamification can create a

Tokuno, K., Kusunoki, F. and Inagaki, S.

Explore Through the Past: Gesture-Based Mobile Game for Children Observing Geological Layer Exhibit at History Museum.

DOI: 10.5220/0011844700003470

In Proceedings of the 15th International Conference on Computer Supported Education (CSEDU 2023) - Volume 2, pages 371-377

ISBN: 978-989-758-641-5; ISSN: 2184-5026

 2023 by SCITEPRESS – Science and Technology Publications, Lda. Under CC license (CC BY-NC-ND 4.0)

371

playful and engaging learning experience through an

interaction style separate from viewing, reading, and

listening. Gamification on vulnerable and

chronologically obscure exhibits in museums enables

us to make complex concepts easier for children to

understand better. It can spark interest and enhance

learning in the museum (Acquah and Katz, 2020).

Satoyama forest management game is a simulation

game that can experience hundreds of years of forest

management. Children understood the concept of

forest managing, overcoming the complexity due to

the scale of time (Yago et al., 2021).

1.2 State of the Art

The use of mobile terminals enables experiencing

gamification in wide areas, such as museums and

outdoors. Location-based mobile game for local

history and cultural heritage sites using widely used

mobile devices (Luiro et al., 2019) have been studied.

However, simply interacting through touch screen

with content displayed on a screen limits the physical

movements that are important in learning.

Learning is more effective when body movements

are involved in the learning process (Gee, 2008).

Embodiment leads to better learning outcomes than in

the non-physical learning in classrooms settings

(Vazou et al., 2019). Integrating physical activities by

asking children to imitate the movement of animals to

had positive effects on children’s understandings and

memory on geology concepts (Mavilidi et al., 2016),

and foreign language vocabulary learning (Schmidt et

al., 2019). With motion tracking sensors such as

Microsoft Kinect, physical movements can be used to

directly interact with digital contents. The Angle-

makers helps children understand angle by tracking

both arms and calculating (Georgiou et al., 2020).

Museums settings have utilized this to introduce full

body interactive installations. By using gestures and

body movements to control exhibits, visitors

experienced more engagement and joy. UKIO-E

game allows children to have conversation with

painters of Japanese traditional arts using gestures

and voices (Tamaki et al., 2016). Belong allows

children to interact with dinosaurs with multiple

gestures (Tokuoka et al., 2017). In both games,

children’s interest significantly increased. However,

learners’ use of gestures in gamification experiences

with mobile devices have not been considered

enough. By incorporating physical actions into game-

based learning using mobile devices, we aim to make

learning more effective while maintain versatility.

In this study, we describe the development of a

mobile gesture-based action game that enables

learners to learn about vulnerable geological

formations through fun and playful physical actions.

Here, we develop a game in which the contents of a

geological strata cross-section are displayed, and the

user can interact with the contents by manipulating

the characters through physical gesture actions. The

proposed system consists of tablet devices alone and

built-in accelerometer was used to achieve the gesture

recognition. Through this system, the learner

observes the geological strata while moving his or her

body. In this study, we focused on the effects of

physical movements on learning, and the experiment

was conducted with such purpose. This paper

describes the outline of the proposed system, the

experiment, and the results of the questionnaire

survey.

2 GAMIFICATION OF

GEOLOGICAL LAYER

EXHIBIT

2.1 Overview

In this study, an action game that displays digital

information and requires physical action is used to

enhance the geological strata exhibit in a history

museum into a highly interactive and exciting

experience.

The game allows students to learn about

geological formations from the bottom to the top

along a chronological sequence of strata formation.

The formation of geological strata often takes several

hundred years, a scale of time that cannot be

experienced in ordinary life. For example, the 4-

meter-high, 6-meter-wide stratum in Figure 1 shows

the lower part of the stratum, the oldest part that dates

back to the 13th century, and the upper part that dates

back to the 17th century.

Figure 1: Geological Strata Exhibit at Museum.

CSEDU 2023 - 15th International Conference on Computer Supported Education

372

The learner climbs a game stage course

reconstructed the strata by controlling a small

character along a chronological sequence from a start

point in the lower layer, which is the older strata, to a

goal point in the upper layer, which is the newer layer.

In the process of climbing, game displays contents

matching the artifacts in the stratum, dialogues of

characters to enhance the immersive experience, and

brief descriptions of the stratum according to the

location of the artifacts. The learner climbs a course

by using the characteristics of the displayed contents

and aims at the goal. The learner can observe the

numbers of digital contents and the artifacts in the

stratum as shown in table 1 at the same time to learn

the chronology of the excavated artifacts and their

positional relationship in the strata. The system setup

is shown in Figure 2.

Table 1: Artifacts in Geological Layer.

Categor

Number of contents

Buildin

Surface 8

Road Gutte

Plate 3

Bone 3

Burnt Soil 4

Figure 2: System Setup.

The game is played by the learner holding the

device and scanning the strata exhibit with the camera.

When the system recognizes the exhibit, a character

and course are superimposed on the exhibit. The

player can control the character by tilting the device,

shaking it up and down, and tapping the screen to

reach the goal. The game experience flow is shown in

Figure 3 and Figure 4.

Figure 3: Experience Flow.

Figure 4: Game Flow.

2.2 Creating Exploration Experience

An understanding of a geological time scale is an

important element in learning about geologic strata.

Children understand the environmental changes of

the Earth in ancient times, but they think in terms of

Explore Through the Past: Gesture-Based Mobile Game for Children Observing Geological Layer Exhibit at History Museum

373

two rough time divisions, “very old age” and “old age”

rather than specific years (Trend, 1998). To simplify

children’s understanding of geological strata, this

game does not use specific ages but divides the strata

into three major layers: the old, middle, and upper

strata. The learner can freely climb from the bottom

to the top of these three layers. The exhibit of the

virtual space is darkened so that the learner can

experience the adventure of exploring the places of

interest to them. The character is shown glowing, and

the excavated artifacts can be seen only by controlling

the character. Figure 5 shows this scene.

Figure 5: Character Lighting up the Geological Layer.

The excavated items contained in the stratum are

difficult to observe with the naked eye because of

their small size compared to the size of the exhibit and

because they have lost their original appearance

because of weathering. For example, Figure 6 shows

an excavated item on display. On the left is a pot made

of clay, and on the right is a dolphin spine. Although

they are labelled, recognizing the items is very

difficult. Especially for children.

Figure 6: Excavated Items Displayed.

For children, the museum experience can be

strongly felt by linking their direct experiences with

the exhibits (Piscitelli & Anderson, 2001). Therefore,

we selected content that local children would find

familiar. For the building surface, we chose artifacts

familiar to the local residents of the city of Fukuoka,

where the museum is located. To make it easier for

the children to recognize, items are illustrated, as

shown in Figure 7.

Figure 7: Illustration Content Shown in AR.

2.3 Climbing up the Layers Using Body

Gestures

This system requires that the device reads the

learner’s movements and translates them into the

actions of the digital character on the screen in real-

time. This allows learners to have an immersive

experience of learning geological strata by

controlling the characters displayed on the strata

through their physical movements. We used the

acceleration sensor value built into tablet devices to

measure the learner’s movement. The acceleration of

the terminal in the z-direction is measured, and when

a tilt of more than a certain amount is detected, it is

converted into the character’s horizontal movement

speed in proportion to the acceleration. As shown in

Figure 8, the character moves horizontally when the

learner holds the device horizontally and tilts it. The

character moves faster in proportion to the tilt of the

device.

The game also includes a jumping motion because

of the nature of the game’s strata-climbing game.

Tapping the screen moves the character to jump

vertically. The longer the tap on the screen, the larger

the jumping motion.

Player

CSEDU 2023 - 15th International Conference on Computer Supported Education

374

Figure 8: Tilting Mobile Tablet to Control Character in

Game.

3 EXPERIMENT

3.1 Experiment Method

The proposed system was evaluated by 24 visitors (17

boys and seven girls), ranging from the first grade of

elementary school to the second grade of high school.

The evaluation was conducted at the Fukuoka City

Museum. Participants were a mixture of siblings and

single visitors, and only one iPad was provided for

each experience. Participants played the game for

around 15 minutes after a brief explanation, and the

completion of the experience was optional. Figure 9

shows the game being played by a single participant

and Figure 10 shows multiple participants playing it

simultaneously.

Figure 9: Single Participant Playing the Game.

Figure 10: Participants in a Group Playing the Game.

First, participants briefly observed the strata.

Thereafter, they moved to a designated position in the

center at the front of the stratum, holding the iPad

with both hands, and experienced the game. The

participants followed the guidance displayed in the

game and freely controlled the characters by moving

their iPads. Participants soon got used to the game

and enjoyed playing it regardless of the number of

people in the group. Finally, after completing the

game, a simple questionnaire was used to evaluate the

game. The questionnaire consisted of 12 questions,

four questions were about the game, three were about

fatigue, four were about the strata exhibit and a free

answer section. Each question was scored on an 11-

point scale from 0 to 10, with 0 corresponding to “not

at all disagree” and 10 to “strongly agree.”

3.2 Results

The responses to the questionnaire are described as

follows: scores from 0 to 3 were classified as negative

responses, from 4 to 6 as neutral responses, and from

7 to 10 as positive responses. The number of positive

responses and the number of neutral and negative

responses were analyzed using a direct probability

computation method. Questionnaire results are

presented in Table 2.

First, the evaluation of the game was summarized.

The following four questions were asked: “Was the

game fun?” “Was the game easy to understand?,”

“Would you like to play more?,” and “Did you feel a

sense of accomplishment?” Positive responses far

exceeded the neutral and negative responses. 23 out

of 24 participants enjoyed the game, and wanted to

keep playing, even though as shown in table 3, some

participants felt the game was a little difficult. Next,

the evaluation of fatigue was summarized by asking

the following three questions: “Does your body feel

tired?” “Does your brain feel tired?,” and “Was the

character easy to control?” Positive responses to these

questions also far exceeded the neutral and negative

responses. Comments such as “It was very easy

because I only need to move my device” show it was

easy to control the character through physical gesture.

Finally, the evaluation of interest in and

understanding of the exhibition was summarized by

asking the following four questions: “Did the game

make you want to know more about the contents?”

“Would you like to see the exhibition again?,” “Can

you imagine the contents?,” and “Do you understand

the chronological order?” Positive responses to these

questions exceeded the neutral and negative

responses. Comments such as “It was easy to

understand, and I thought people who do not know

Explore Through the Past: Gesture-Based Mobile Game for Children Observing Geological Layer Exhibit at History Museum

375

about history can also enjoy this.,” “I understood

Fukuoka city’s history” younger participants believed

they have gained knowledge from playing this game.

Table 2: Results of Questionnaire.

Question Positive Neutral Negative

Was the game fun? 23 1 0

Was the game easy to

understand?

23 1 0

Would you like to play

more?

23 1 0

Did you feel a sense of

accomplishment?

23 0 1

Does your body feel

tired?

21 1 2

Does your brain feel

tired?

14 6 4

Was the character easy

to control?

15 2 7

Did the game make

you want to know

more about the

content?

23 0 1

Would you like to see

the exhibition again?

19 5 0

Can you imagine the

contents?

19 5 0

Do you understand the

chronological order?

17 6 1

Table 3: Free answer field.

Age Answe

7 It was fun.

8 It was fun.

9 It was entertaining.

9 It was very easy to understand and very fun.

Jumping was hard, and when I succeeded, I was

very happy.

It was easy to understand, and I thought people who

do not know about history can also enjoy this.

10 I understood Fukuoka city’s history.

It was very easy to control the character. I want

difficulty level selection.

It was very easy because I only need to move my

device.

11 It was fun.

12 It looked like a real thing.

It was interesting, I thought there was a problem

with the wall-jumping.

13 It was difficult but fun.

13 I was hoping for something a little easier.

13 It was difficult but interesting.

14 It was interesting.

I liked the fact that I was able to have fun and

cooperate with my friends when we got through the

fire part.

It was interesting because of the special way of

operation, but I didn’t gain much knowledge.

The results of this study suggest that the use of

gesture-based learning experiences can lead to an

interest in geological formations and make learning

about geological formations in museums more

effective way. The nature of the game, which allowed

children to acquire information about the exhibit by

freely exploring areas of interest sparked children’s

interest toward geological strata exhibit, motivating

children to learn about the geological strata exhibit

more, and led to understandings of basic geological

formations.

4 CONCLUSIONS

This paper describes the development and evaluation

of a game for geological layer exhibit to support geo-

science education in a museum. For children to enjoy

and learn geological layer exhibit effectively, we

developed a game in which children can learn about

geological layer by controlling digital characters

through a mobile tablet device with physical gestures.

The effectiveness of the system was evaluated

using a questionnaire consisting of 11 questions.

Positive responses exceeded neutral and negative

responses for all questions related to the game, fatigue,

and interest in geological formations. The children

found the game engaging and educational and were

willing to try it again. The game was easy to use and

natural, suggesting that the game increased interest in

the exhibits. The results of this study suggest that the

learning through exploration with use of gesture-

based learning experiences can lead to an interest in

geological formations and make learning about

geological formations in museums more effective.

These results establish the possibility of an education

using simple gesture motion that provide children

with a better play and learning experience.

However, this limitation has been identified in

this system. Area based AR marker can be unstable at

areas with insufficient light sources. To stabilize the

AR at such areas, placing AR markers are desirable.

Therefore, in future work, we would like to stabilize

the system by placing multiple AR markers on the

floor. The goal of this research is to develop an

effective system for learning with mobile terminals in

museums using body movements. With such a system,

we can make museums more enticing and effective

for children’s knowledge acquisition.

CSEDU 2023 - 15th International Conference on Computer Supported Education

376

ACKNOWLEDGEMENTS

This work was supported by JSPS KAKENHI Grant

Number 22H00078, 22H01068.

The evaluation experiment was supported by The

Fukuoka City Museum.

REFERENCES

Acquah, O. E. Katz, H. T. 2020. Digital game-based L2

learning outcomes for primary through high-school

students: a systematic literature review. Computers &

Education, Volume 143, 103667. doi: 10.1016/

j.compedu.2019.103667

Arima, T. 2016. Overview: The educational capabilities of

geoparks: From education to learning. Journal of

Geography (Chigaku Zasshi), 125(6), pp. 775-778. doi:

10.5026/jgeography.125.775

El Mawas, N., Truchly, P., Podhradský, P. Muntean, C.

2019. The effect of educational game on children

learning experience in a slovakian school.

In Proceedings of the 11th International Conference on

Computer Supported Education (CSEDU), Volume 1,

ISBN 978-989-758-367-4; ISSN 2184-5026, pp. 465-

472. doi: 10.5220/0007764104650472

Ford, D.J. 2005. The challenges of observing geologically:

Third graders' descriptions of rock and mineral

properties. Sci. Ed., 89, pp. 276-295. doi: 10.

1002/sce.20049．

Gee, J. P. Learning and games. The ecology of games:

connecting youth, games, and learning. 2008. Edited by

Katie Salen. The John D. and Catherine T. MacArthur

Foundation Series on Digital Media and Learning.

Cambridge, MA: The MIT Press, pp. 21-40. doi:

10.1162/dmal.9780262693646.021

Georgiou, Y., Ioannou, A. 2021. Developing, enacting and

evaluating a learning experience design for technology-

enhanced embodied learning in math classrooms.

TechTrends, 65, pp. 38-50. doi: 0.1007/s11528-020-

00543-y

Gerven, V. D., Land-Zandstra, A., and Damsma,

W. 2018. Authenticity matters: Children look beyond

appearances in their appreciation of museum

objects, International Journal of Science Education,

Part B, 8:4, pp. 325-339. doi: 10.1080/21548455.2018.

1497218.

Henderson, T.Z., and Atencio, D. J. 2007. Integration of

play, learning, and experience: what museums afford

young visitors. Early Childhood Education Journal,

35(3), pp. 245-251. doi: 10.1007/s10643-007-0208-1.

Luiro, E., Hannula, P., Launne, E., Mustonen, S.,

Westerlund, T., and Häkkilä, J. 2019. Exploring local

history and cultural heritage through a mobile game. In

Proceedings of the 18th International Conference on

Mobile and Ubiquitous Multimedia (MUM '19).

Association for Computing Machinery, New York, NY,

USA, Article 42, pp.1–4. doi: 10.1145/3365610.

3368411

Mavilidi, M.-F., Okely, A.D., Chandler, P. and Paas, F.

2016. Infusing physical activities into the classroom:

rffects on preschool children's geography learning.

Mind, Brain, and Education, 10, pp. 256-263. doi:

10.1111/mbe.12131

Piscitelli, B. and Anderson, D. 2001. Young children’s

perspectives of museum settings and

experiences. Museum Management and Curatorship

19, no. 3, pp. 269-282. doi: 10.1080/

09647770100401903

Schmidt, M., Benzing, V., Wallman-Jones, A., Mavilidi, M.

-F., Lubans, D. F., and Paas, F. 2019. Embodied

learning in the classroom: Effects on primary school

children’s attention and foreign language vocabulary

learning, Psychology of Sport and Exercise, Volume

43, ISSN 1469-0292, pp. 45-54. doi: 10.1016/j.

psychsport.2018.12.017

Tamaki, H., Sakai, T., Ota, Y., Egusa, R., Inagaki, S.,

Yamaguchi, E., Kusunoki, F., Namatame, M.,

Sugimoto, M., and Mizoguchi, H. (2016). Participatory

design of UKIYO-E game for children to support art

appreciation based on interacting with pictures. In

Proceedings of the The 15th International Conference

on Interaction Design and Children (IDC '16).

Association for Computing Machinery, New York, NY,

USA, pp. 637–642. doi:10.1145/2930674.2936007

Tokuoka, M., Tamaki, H., Sakai, T., Mizoguchi, H., Egusa,

R., Inagaki, S., Kawabata, M., Kusunoki, F., and

Sugimoto, M. 2017. BELONG: body experienced

learning support system based on gesture recognition -

enhancing the sense of immersion in a dinosaurian

environment. In Proceedings of the 9th International

Conference on Computer Supported Education

(CSEDU), Volume 1, ISBN 978-989-758-239-4; ISSN

2184-5026, pp. 487-492. doi: 10.5220/0006357

104870492

Trend, R. 1998. An investigation into understanding of

geological time among 10- and 11-year old children.

International Journal of Science Education, 20, pp.

973-988. doi: 10.1080/0950069980200805Moore

Vazou, V., Pesce, C., Lakes., K and Smiley-Oyen., A. 2019.

More than one road leads to Rome: A narrative review

and meta-analysis of physical activity intervention

effects on cognition in youth. International Journal of

Sport and Exercise Psychology, 17:2, pp. 153-178. doi:

10.1080/1612197X.2016.1223423

Yago, K.; Shingai, Y.; Kobayashi, W.; Aoki, R.; Takeda,

Y.; Kusunoki, F.; Mizoguchi, H.; Sugimoto, M.; Funaoi,

H.; Yamaguchi, E. and Inagaki, S. 2021. Satoyama

forest management learning game for SDGs education:

comparing the effect of providing additional

information in the first half and latter half of the game.

In Proceedings of the 13th International Conference on

Computer Supported Education (CSEDU), Volume 1.

ISBN 978-989-758-502-9; ISSN 2184-5026, pp. 347-

351. doi: 10.5220/0010473703470351.

Explore Through the Past: Gesture-Based Mobile Game for Children Observing Geological Layer Exhibit at History Museum

377