IF ONLY...

Designing Novel Games for Collaborative Learning

with Smartphones and Sensors

Cliff Randell

University of Bristol, Woodland Road, Bristol, U.K.

Ben Clayton

Calvium Ltd., Pervasive Media Studio, Anchor Square, Bristol, U.K.

Tarim

Media Playgrounds, Pervasive Media Studio, Anchor Square, Bristol, U.K.

Tom Bennett

Interactive Places, Pervasive Media Studio, Anchor Square, Bristol, U.K.

Keywords:

Mobile learning, Play, Games, Smartphones, Handheld computers, Sensors, Actuators, Climate change.

Abstract:

In this project we describe research into playful learning experiences featuring smartphones and external

sensors. We built and tested a number of games which can be played using environments populated with

sensors and/or actuators interfaced wirelessly with smartphones. An overall story interlinking the games was

developed based on the need for action to tackle climate change. Trials with teenage schoolchildren revealed

approaches which both appeal to players and have the potential to educate.

1 INTRODUCTION

Learning as you play has been recognised as a valid

technique, particularly for children, for many years. It

is well researched and documented (Piaget, 2002; Pa-

pert, 1993). The brain loves to learn patterns as they

enable us to be efﬁcient and thereby survive. Once

a pattern has been seen in something, and has been

understood, we can move on to the next challenge.

When presented with something new, we automati-

cally try to see already recognised patterns in it so that

we can create a new chunk, and move on. This ability

to chunk and reuse knowledge provides a mechanism

for transferring experience learnt from playing games

to real life.

Games are abstracted versions of the world, and

are therefore ‘safe’. There may be risk/jeopardy

within a game but it has no impact on the ‘real’ world

and trial and error can be used to learn the patterns

of play. In formal learning being good at games is

not considered as important as being good at reading,

writing and mathematics, and yet the brainis perform-

ing exactly the same routine. A good game is one

that has the right balance of challenge and achieve-

ment. Too difﬁcult and players give up. Too easy

and players get bored. Get it right, and the brain goes

into ‘ﬂow’ mode releasing endorphins and making

the learning experience pleasurable (Csikszentmiha-

lyi, 1975).

With the advent of modern mobile computer tech-

nology forms of play are changing rapidly to include

interaction with virtual scenarios using smartphones.

These can provide alternate worlds or act as media-

tors with the real world. Consequently expectations

of ‘play’ are changing. In this project we use two dif-

ferent approachesto explore the appeal of a number of

futuristic playful learning games using smartphones.

The use of handheld computers for educational

purposes outside of the classroom has already

taken many forms. For example, in the Ambient

Wood project an outdoors learning experience for

schoolchildren was created in a Sussex woodland

207

Randell C., Clayton B., . T. and Bennett T..

IF ONLY... - Designing Novel Games for Collaborative Learning with Smartphones and Sensors.

DOI: 10.5220/0003303702070212

In Proceedings of the 3rd International Conference on Computer Supported Education (CSEDU-2011), pages 207-212

ISBN: 978-989-8425-49-2

 2011 SCITEPRESS (Science and Technology Publications, Lda.)

aimed at encouraging students to carry out contex-

tualized scientiﬁc enquiry and to reﬂect on their in-

teractions (Rogers et al., 2005); and in the Savannah

project players experience and learn collaboratively

about a virtual savannah that appears to be overlaid on

a grassy ﬁeld (Benford et al., 2004). The introduction

of competitive gaming has also been shown to have

educational beneﬁts (Di Blas et al., 2010). In our

project we are interested in creating interlinked, in-

teractive scenarios which provide schoolchildren with

challenges which are both enjoyable and educational.

We use smartphones and external sensors to provide a

cohesive environment in which a storyline can be de-

veloped. The smartphones are an integral part of each

of the scenarios.

The underlying objective is to create cross-

platform, smartphone learning game applications

which will work with a number of sensors. This ap-

proach enabled us to demonstrate and streamline the

production process and skills required in creating a

novel product and experience, crossing ﬁlm, game

and online experiences with physical interaction with

real objects. The project shows how sensor technol-

ogy can be integrated with existing smartphone tech-

nology to create new, exciting and dynamic learning

experiences.

This paper describes two approaches using cli-

mate change to provide the underlying scenario. The

players are presented with a series of challenges

which if completed successfully will give them the

arguments necessary to change the development strat-

egy of a power generation company, headed by ‘Bran-

don Riches’. The players are guided by a girl from the

future, ‘Gaia’. The ﬁrst approach is described in sec-

tion 2 where fact ﬁnding challenges are completed by

using iPhones. In the second approach, described in

section 3, the challenges are more physical with addi-

tional gameplay. Section 4 gives a technical overview

and we conclude in section 5.

2 FACT FINDING GAMES

The ﬁrst iteration of the project was intended to raise

awareness of climate change issues by providing sim-

ple games in which iPhones gather relevant facts us-

ing data triggers. Before any application building was

started a workshop was held with a group of 12 to

14 year olds in order to test the game concepts. We

wanted to be sure that the concepts of the story and

the games were appealing to the target audience.

Using feedback from the workshop we designed a

set of climate change related games linked together

using the storyline which involved Gaia reporting

from a climate devastated future using a time-shifted

video link. The overall object was to collect facts to

change the investment strategy of Brandon Riches,

our power magnate. The games were introduced by

Gaia. Each game followed the same procedure: enter

game region; receive video call; ﬁnd source of game

instructions (video tells you how); listen to/watch in-

structions; complete game; receive score and store in-

formation gained; and ﬁnally move onto the next re-

gion. The games were intended to be played indoors

or outdoors using a mixture of WiFi and GPRS to pro-

vide connectivity to a central server.

Six themes were chosen and different sensing

technologies were associated with each one.

• Transport - To complete the game the player had

to cycle a ﬁxed distance on an exercise bike. The

game was started by ringing the bicycle bell.

• Clothes - The players identiﬁed sustainably pro-

duced items from a range of clothes on display

using a sonar-like system. The iPhone emitted a

pulsed audio tone which was received by hidden

microphones in the relevant clothes. A shop bell

was used to initiate the game.

• Deforestation - By showing the iPhone display to

a tree ﬁtted with a webcam/light sensor, informa-

tion is revealed on the iPhone when the display is

recognised.

• Travel - In response to correctly answered mul-

tiple choice questions on the iPhone, UV painted

signs are illuminated by UV lights.

• Energy - Strategically placed QR codes are asso-

ciated with different energy sources. The iPhone

reads the codes, causing a call to be made to the

iPhone in which the information is given as a ver-

bal message.

• Food - In a similar design to the Energy game, bar

codes on food products are read by the iPhone.

The games were followed by a video sequence

in which Brandon Riches explained the affect of the

gathered information on his thinking.

A group of sixteen 14 year olds from a local

school were invited to carry out a trial of the game.

They participated in pairs, each pair using a single

iPhone or iPod touch. This was intended to encourage

collaboration. It was clear that the games with multi-

ple components were the most popular, and any rep-

etition of task spoiled the enjoyment of doing some-

thing new; the patterns were easily learnt. The Trans-

port and the Energy games were enjoyed more than

any others. The fact that these were both the most en-

joyable and very different from one another caused us

to reconsider some of the other sensors we intended

using. Often in games, players like the opportunity

to skill-up through repetition, but this age group ap-

CSEDU 2011 - 3rd International Conference on Computer Supported Education

208

peared to relish the new challenges that each game

brought. This may have been a desire to grasp and

complete each new concept as quickly as possible.

In the feedback session the children reported that

the aspects they liked best were the active/physical

games; the searching for QR codes and items of

clothing; ease of use; continual feedback; games that

worked and watching each other playing the games.

Technical problems which had a negative impact on

their experience included using headphones (espe-

cially splitters for dual use); having to change hand-

sets (as we had a mix of iPhones and iPod touches);

and games/sensors apparently not working. They

also found the games too easy. Useful points were

also made on feedback sheets: “may be better for a

younger audience”, “this age prefer guns and adven-

ture”, “a platform game that requires skill to progress

would be better”, “avoid anything educational as it’s

too preachy and we’ve heard it all before”!

Enjoyment was possibly impaired by issues of in-

terconnecting devices using WiFi, GPRS etc, as some

of them were almost certainly dropping connections

as they left/entered the building. Also the fact that

videos seemed to stop when headphones were re-

moved was an issue. Some of the players didn’t re-

alise that the real bell had to be rung in the ‘shop’,

and at times it would have been better to have audio

instructions as well as text, with the ‘continue’ button

only appearing on the screen when the audio ﬁnishes,

so that it would not be possible to skip forward with-

out hearing important instructions. Coupled with this

would be more ‘back’ buttons, and the option to quit

if the game is taking too long.

On the basis of the feedback we decided on a rad-

ical re-think of the project. If we wished to retain the

prime emphasis of the story promoting the need to

tackle climate change then we would need to switch

to a younger audience. Alternatively we could make

major changes, for instance switching game and story

in importance so that the games become the focus,

not the story and theatrical elements. Either way we

needed to improve the reliability of the experience.

3 ‘FUN’ GAMES

3.1 Redesign

We therefore decided to make some major changes

with the emphasis on games which provided a greater

challenge to the player with secondary educational

beneﬁts, and also to introduce an element of fun. The

story became the means to connect the games. We

decided to stay with the mid-teens age group bear-

ing in mind the difﬁculty of producing something

that would be enjoyed by a wider age group. So we

worked on games which were ‘fun’, incorporating the

iPhone with external sensors, and then tried to weave

the games into the climate change story. To improve

reliability we chose to use WiFi for as many games

as possible, to abandon the use of headphones, and to

restrict the overall game area to a number of indoor

rooms with space made available for a small audience

to view each game as it was being played. We could

then concentrate on improving the games.

The games that were developed were intended to

be collaborative with ideally two players, and also be

entertaining to watch. Each game would provide so-

lutions to parts of an overall puzzle with top scorers

winning prizes. The games could be played more than

once if desirable. The structure previously developed

of a video introduction to each game, and an initiat-

ing event were retained. Game instructions were also

shown on-screen to assist the players and a menu used

to select each game (see Figure 1).

Figure 1: iPhone Menu.

After a brainstorming session, the following

games were developed and tested with colleagues:

• Bike Charge. An extension of the bicycle game

described in Section 2 with improved feedback on

the iPhone. As before ringing the bicycle bell ini-

tiated the game. One player then had to cycle to

three (virtual) checkpoints over set distances, with

each becoming harder. Reaching each checkpoint

earnt the player points and started virtual wind tur-

bines shown on the iPhone. The second player

used the iPhone to report progress, and encourage

the ﬁrst. A ‘pause’ button enabled the players to

swap roles.

• Code Clap. A grid of nine squares was laid out

on the ﬂoor, with key numbers in them to align

the grid with a numbered diagram on the iPhone

screen. A single handclap initiated the game. One

IF ONLY... - Designing Novel Games for Collaborative Learning with Smartphones and Sensors

209

player held the iPhone and observed the squares

lighting up in sequence on the display and the

other player had to move to the associated square

on the ﬂoor and perform a handclap. The game

sped up until the code had been completed.

• Hidden Symbols. This game was played in a

dark room, and was initiated by opening the door.

Three correctly answered multiple choice ques-

tions on the iPhone reveal three hidden painted

symbols on the walls by turning on UV lights.

These symbols then have to be recognised from

a list of symbols on the iPhone.

• Corridors of Power. A novel interface was de-

signed using two proximity sensors. Movement

of the players hands - left, right or both - were

sensed to enable the player to navigate through a

3D maze. The maze depicted corridors in a power

station where ‘power rings’ could be collected.

The other player viewed a 2D blueprint version

of the maze displayed on the iPhone and was able

to give directions to the next ring. A game being

played is shown in Figure 2. Crossing a start line

initiated the game, and players had to collect six

rings and then exit the maze.

• Short Circuit. In this game, the iPhone was used

to control three motors which move indicator ﬂags

on a display panel. The motors run at different

speeds and with some latency. The player was re-

quired to line up the ﬂags. This is a speed/skill

test with the accelerometers in the iphone being

used to activate the motors when the iphone was

tilted.

• Power Cut. Again using actuators, this game in-

tended to use mirrors mounted on servomotors

controlled by the iPhone. These would to direct

a laser beam around a room onto a light sensor.

Completion of each game gave the players part

of the solution to an overall puzzle. Initial feedback

from colleagues was positive with the comparison be-

ing made with a fairgroundexperience combining dif-

ferent attractions into a cohesive experience. The use

of the iPhone as a gateway to, and not as the focus

of, the experience was also recognised as a welcome

feature.

3.2 Testing

For our ﬁnal testing, we invited the previous group of

school pupils to return. The six games were presented

in an interface shown after an introductory video from

Gaia. The players could choose which one to do from

this menu as shown in Figure 1. The games all fol-

lowed a similar pattern, with instructions on where to

go and what to do when there to get instructions, both

Figure 2: Playing in the Corridors of Power.

on video and onscreen as text.

The ﬁnal game Power Cut was not completed as

there was not the time to overcome the accuracy prob-

lems, although it would have been relatively simple to

include as it used the same software as Short Circuit.

This was largely due to the fact that Short Circuit pre-

sented many mechanical issues to the extent that it

was still not working reliably enough when the pupils

arrived. Disappointingly, then, we only tried out four

of the games on the pupils. They all worked as in-

tended, although some did throw up other issues.

The games were well received by the 16 partic-

ipants (10 boys, 6 girls, age 14-15) and both verbal

and written feedback was collected. The favourite

games were Corridors of Power (8 votes); Code Clap

(4 votes) and Hidden Symbols (3 votes). The boys

particularly liked the Corridors of Power, and the girls

particularly disliked Bike Charge.

A series of questions were asked as part of a short

questionnaire given to the participants after the test

session. For each question the player had to provide a

rating on a graphic scale, i.e. make a mark on a 10cm

line reﬂecting their agreement from 0 (not at all) to

10 (very much). From these marked scales we learnt

that:

- all the players found the instructions easy to follow;

- nearly all enjoyed the experience very much;

- three found it too easy, but still enjoyed it very much;

- they all enjoyed the Corridors of Power;

- the girls liked the Code Clap and Hidden Symbols,

but only a couple liked Bike Charge. Three really

didn’t like the Bike game at all;

- the boys tended to be more neutral, with the Bike

Game getting more scores towards the top of the

scale. One boy didn’t like Code Clap, and one boy

really didn’t like the Hidden Symbols game.

We also enquired which age range they thought

would be appropriate for this experience. Three said

10-12, and ﬁve more had the lowest age as 10, but

CSEDU 2011 - 3rd International Conference on Computer Supported Education

210

apart from the three, all had mid-teens as their range

(with some going higher).

In discussion they commented that overall it was

much better than the previous time, though one player

did enjoy playing the games outside during the pre-

vious tests. They thought that the games were “fu-

turistic”, “more modern than most games”, “unique”,

“more active than most games” and simply “fun”.

Further exploration revealed that they thought that it

was better because the games were interlinked and

made sense within the story. They all agreed it was

challenging and fun both to watch and to do.

4 TECHNICAL OVERVIEW

4.1 Architecture

The If Only game used a central Notiﬁcation Server

to keep track of which iPhones are interested in which

events. A Variable Database was also provided on

the server to store the current state of the system. An

Event Client ran on each of the iPhones updating the

User Interface when events were triggered remotely

by sensors; and also sending events to the Notiﬁca-

tion Server to operate actuators. Communication was

enabled by using HTTP over WiFi or GPRS to/from

the server. These modules are described in more de-

tail below:

Variable Database. For each possible event in the

game, a named variable was created and stored in this

database. The current value of the variable and the

time the variable was last modiﬁed were stored. Up-

dates to the value of the relevant variable were made

using an HTTP request containing the variable name

and value to the server.

Event Client and Notiﬁcation Server. The Event

Client runs on the iPhone and is used to allow the user

interface to be updated when the value of particular

variables change in the database. For instance, when

the Bike Charge game is started, the iPhone sends

a message (using HTTP) to the Notiﬁcation Server

asking to be notiﬁed when the ‘bikeBell’ variable is

changed. When this change happens, the iPhone plays

a sound and switches the user interface to the ‘start

pedalling’ game screen. This was achieved by using

long polling where the server will deliberately not re-

turn a response until an event of some kind has hap-

pened. This technique means that the iPhone is able

to respond to changes usually in less than 200ms over

WiFi.

Actuation System. The actuation system was used

to cause an action in the real world as a result of per-

forming an action on the device screen, such as an-

swering a question or pressing a button. The iPhone

software itself would make an HTTP request to the

Actuation Server with the name of the device to actu-

ate, and the desired value.

4.2 Sensors and Actuators

A diverse collection of sensors and actuators were

employed using bespoke PIC and Arduino microcon-

troller boards to interface with wireless internet capa-

ble devices (Microchip, 2010; Arduino, 2010). The

data was sent to the server either via an Arduino and

GPRS internet connection, or via a Macbook using

WiFi. The devices included:

• Transport/Bike Charge. A magnet mounted on

the bicycle wheel operated a reed switch on the

frame to sense the rotation of the wheel and hence

determine the distance travelled. The bicycle bell

was ﬁtted with a microswitch. Both switches were

connected to an Arduino with a GPRS shield.

• Sonar. The iPhone generated a high-pitched

beeping sound which was picked up by the hidden

wireless microphones. The signal was ampliﬁed,

ﬁltered and sampled to determine the proximity of

the iPhone to the microphone. The shop bell was

ﬁtted with a vibration sensor. The sensors were

connected to an Arduino.

• Deforestation. A simple light detector at the end

of a long tube was used, again connected to an

Arduino.

• Travel/Hidden Symbols. A commercial mains

remote control was adapted and received con-

trol signals from an Arduino. The Remote con-

trol switched three UV ﬂuorescent lamps on and

off. The door to the room was ﬁtted with a mi-

croswitch.

• Energy and Food. The built in iPhone camera

was used to read the QR and bar codes.

• Code Clap. Four 25kHz ultrasonic sensors were

arranged in a square on the ceiling of the playing

area. By measuring the arrival times of the hand-

claps at the sensors, an estimate of the position of

the claps was obtained using a PIC and MacBook.

• Corridors of Power. The two handed proxim-

ity detector was built using commercial Theremin

modules (IEEE, 2008; Doepfer, 2010) and a PIC

adaptor. These were connected to a MacBook

which rendered the maze and provided sound ef-

fects. The MacBook connected to the server using

WiFi, sending the current position and orientation

of the player.

• Short Circuit and Power Cut. DC motors and

servos were controlled by an Arduino.

IF ONLY... - Designing Novel Games for Collaborative Learning with Smartphones and Sensors

211

Apart from the UV lights, all the sensors and ac-

tuators were battery powered. Observations on the

technical performance of the architecture, sensors and

actuators are included in the next section.

5 CONCLUSIONS AND FUTURE

WORK

The If Only ... project has tested two very differ-

ent approaches to playful learning using smartphones

and sensors. In the ﬁrst the emphasis was placed on

learning about climate change through gathering facts

and considering their implications. Sensors were used

as triggering devices to reveal pertinent information.

The second approach was to concentrate on produc-

ing ‘fun’ games with a greater emphasis on challenge

with the learning process being incidental. In this case

sensors and actuators were used as integral parts of

interactive games. This approach clearly had many

beneﬁts and was enjoyed more by the test group, with

enthusiasm shown to return to the games. Taking the

focus of the games away from the smartphone and

using it as a gateway to a larger experience worked

well. Important facets of active learning were demon-

strated with the collaborative aspect generating dis-

cussion and deepening learning, and the physical as-

pect aiding memory by engaging the players. The lat-

ter being especially important to kinesthetic learners.

Overall the project was successful and much was

learnt about producing games using the iPhone with

external sensors. The subject choice was too well

known, as observed by the schoolchildren, to warrant

any testing of knowledge retention. We would wish to

choose a different topic for a future version in order

to be able to carry out appropriate tests. Using sen-

sors as data triggers was not considered sufﬁciently

novel by the players who relished entirely new forms

of interaction. A future version would also attempt to

increase the learning element while retaining the ‘fun’

aspects of the games. There would also be a greater

element of jeopardy introduced such as a (virtual) se-

curity guard constantly pursuing the players. Apart

from the Hidden Symbols, the games were set up in

empty rooms and would have beneﬁted from more at-

tention to the environment to increase the sense of im-

mersion. The latency associated with GPRS meant

that it had to be used with care, and games requiring

fast interaction needed to use WiFi. The software ar-

chitecture performed well for the limited experience

which was offered, however it would not scale well to

very large numbers of players and games.

The approach of focusing on the games and sen-

sors rather than the story has provided a framework

which has the potential to underpin a range of future

possibilities providing a good mix of familiarity and

surprise in a learning environment.

ACKNOWLEDGEMENTS

Thanks ﬁrstly to the UK Technology Strategy Board

for commissioning this project. Hazel Grian for

story writing and game development. Olice produced

all the video, with Jem Noble for the audio. Ellie

Foreman-Peck designed all the iPhone user interfaces.

Thanks also to the other residents of the Pervasive

Media Studio and especially to the staff and pupils

of St Mary Redcliffe and Temple School who partici-

pated so enthusiastically in the testing.

REFERENCES

Arduino (2010). Mega. http://www.arduino.cc. Retrieved:

29 July 2010.

Benford, S., Rowland, D., Flintham, M., Hull, R., Reid, J.,

Morrison, J., Facer, K., and Clayton, B. (2004). ”Sa-

vannah”: Designing a location-based game simulating

lion behaviour. In Proceedings of Advanced Computer

Entertainment, ACE 2004. ACM Press.

Csikszentmihalyi, M. (1975). Beyond Boredom and Anx-

iety: Experiencing Flow in Work and Play. Jossey-

Bass, San Francisco.

Di Blas, N., Paolini, P., and Torrebruno, A. (2010). ”Inno-

vative technologies and education”. In Proceedings of

the 2nd International Conference on Computer Sup-

ported Education, pages 341–348. SciTePress.

Doepfer (2010). A-178. http://www.doepfer.de. Retrieved:

29 July 2010.

IEEE (2008). IEEE Global History Network: The

Theremin. http://www.ieeeghn.org/wiki/index.php/

Theremin. Retrieved 28 July 2010.

Microchip (2010). Pic 16F872. http://www.microchip.com.

Retrieved: 29 July 2010.

Papert, S. (1993). Mindstorms: Children, Computers and

Powerful Ideas. Basic Books, New York, second edi-

tion.

Piaget, J. (2002). Play, Dreams and Imitation in Childhood.

Routledge, London, ﬁfth edition.

Rogers, Y., Price, S., Randell, C., Stanton, D., Weal, M.,

and Fitzpatrick, G. (2005). Ubi-learning integrates in-

door and outdoor experiences. Communications of the

ACM, 48(1):55–59.

CSEDU 2011 - 3rd International Conference on Computer Supported Education

212