‘Trapped’: An Interactive Gamified Audiovisual Installation

Patrick G. Hartono

, Bayu P. Dirgantoro

Binus International University, Jakarta, Indonesia

Keywords: Gamification, Audiovisual, Installation, Game Technology, Auditory Perception, Ambisonic, Binaural

Abstract: “Trapped” is a site-specific gamified audiovisual installation that aims to unify sonically the player and virtual

environment by implementing the sonic pathfinder method into the underlying concept and interactivity

process, so the player is no longer navigate only with his/her visual cognition but also through the auditory

perception that employed to determine the location of the enemy object. These involve the investigation of the

artistic and interactive elements of computer game using the three-dimensional game engine (unity3D) and

three-dimensional sound engine (google resonance audio) to create a virtual 3D environment of the gamified

audiovisual installation.

1 INTRODUCTION

Trapped is developed to investigate how the auditory

perception can be used as an alternative method for

the player in navigating the virtual 3D environment

within the context of the gamified audiovisual (GA)

installation. Although there already been a few

existed GA works that share similar interests, most of

them are in the forms of music composition and

performance, and nearly none have further

investigated or elaborated the sonic interactivities

(hearing) at the center of the artistic and creative

development.

Upon this notion, we made a 'revolutionary'

decision by presenting the enemy character through

the embodiment of the sonic objects that can only be

located by listening to its sound direction. To realize

this idea a 3D sound engine used to replicate the

spatial character of the real sonic environment so the

player will experience the spatial sound sensation that

travels in multi-direction in conjunction with the

random locations of the enemies within the virtual 3D

environment.

The environmental design is divided into three

different stages. Each level is created based upon the

“general” hearing capability since, in most of our

daily activities, the accuracy of the eye has

overshadowed the importance of the ear yet it is the

ear which is in constant vigil in monitoring our

environment (Hickson&Newton,1981). Gamification

concept, spatial sound, as well as the interactivity

concept of the computer game is discussed to

establish a solid foundation in exploring the sonic

element in correlation on how does the player-

perceived it. However, the explanation of auditory

perception, methodology, and techniques, as well as

multimodal interactivity that fall under the context of

phycology and psychoacoustic, are beyond the scope

of this paper. Trapped is the first artistic outcome of

ongoing research that focuses on the gamified

audiovisual area, where the performative and musical

element will further investigated on the upcoming

research development.

2 RELATED WORKS

2.1 Putney “K” for Game-Audio

Ricardo Climent composed and designed interactive

media work Putney “K” for Game-Audio with the

underlying concept of the navigation of the

intersections between the real, the virtual and the

augmented through the medium of sound, aiming to

construct a new “extended reality” (Climent, 2014).

This concept was further developed as a

collaborative work that produced two outcomes: 1)

Putney K for live game audio and, 2) AR_VR_Putney

an interactive media composition. The classic virtual

synthesizer of 1969 VCS3 is created using a graphical

game engine and featured in both results. The concept

of sonic pathfinder is evolved in Putney K for live

game audio through incorporated of the live

performance of an original EMS VCS3 analog

synthesizer by played Mark Pilkington, and the

Hartono, P. and Dirgantoro, B.

‘Trapped’: An Interactive Gamiﬁed Audiovisual Installation.

DOI: 10.5220/0008526600560061

In Proceedings of the 1st International Conference on Intermedia Arts and Creative Technology (CREATIVEARTS 2019), pages 56-61

ISBN: 978-989-758-430-5

virtual game-audio version played live by Climent.

AR_VR_Putney is an interactive media composition

that employed as the language and grammar for

extended realities; a term which explores augmented

with virtual reality through sound (Climent,

Pilkington and Mesárošová, 2016), which intended to

be presented alongside the original concert version in

the separate venue at the same event. AR_VR_Putney

created through collaboration with architect Alena

Mesárošová and visual artist Manuel Ferrer

Hernández.

2.2 GAPPP – Gamified Audiovisual

Performance and Performance Practice

Artistic based research leads by Marko Ciciliani that

aims to investigate the aesthetic and performative

effect of elements from computer games in the

context of audiovisual composition (Ciciliani, &

Luneburg, 2018) through series interviews of the

related works. Within this first phase of research, the

researcher is focused on the exploration of the game

element in the context of audiovisual composition and

their perception by the audience, in which the impact

of the game system on performers and how their

experience translates to an audience will be the

objective of the second phase.

At the time of this paper, Ciciliani had composed

three gamified audiovisual compositions as the

artistic outcomes of the research: 1)"Kilgore" for two

performers and a game system, 2) Atomic Etudes", a

game-based work for Monome and SuperCollider

and, 3) Tympanic Touch" by Marko Ciciliani, a

multisensory work for two performers and a game

system.

3 IMPLEMENTATION

3.1 Narrative Concept

Regardless sophistication of VR and AR technology,

the multidimensional environment of computer game

and our physical world is a definite two dissimilar

reality; whatever happens to the player's avatar in

Gameworld, nothing harmful will occur to the player.

According to (Simon, 2004), when we compose a

narrative, especially a narrative based on memory, we

usually try to represent “how things came to be what

they are,” and the end is prefigured in the beginning.

Simon’s notion revealed the possibility of

empirical communication between the player and

computer game through the stimulation of a particular

memory, which certainly alongside with the cognitive

input of game controller in any forms.

Therefore, the narrative concept of trapped is

"sonic unification," i.e., to "unify" sonically the

player and the Gameworld by implementing the

auditory stimulation and sonic pathfinding at the

center of interactivity process.

3.2 Storyline Concept

The concept of paracusia or auditory hallucination is

employed to develop the narrative concept further,

which ultimately composed a storyline idea as

follows: ‘E’ is the main game character that suffers

from paracusia which incapable of distinguishing

between the real and imaginary sound.

She trapped inside an imaginary world (maze-

like) that consist of three-game stages with a different

auditory obstacle (enemy) on each stage. To identify

the enemy location, she must listen to its sound

direction and colliding herself to the object to destroy

it, by doing that, she earns the rewards to get into the

next stage. The game-stages design and perceptual

appearance of the enemy have been adapted

according to the narrative concept and will be

discussed further on the next sections of the paper.

3.3 Perceptual Appearance of The Enemy

Object

In a general computer game, the enemy object is

presented through the commons antagonist character

that can visually be identified by the player. However,

due to the implementation of sonic pathfinding, the

common perception of space (visual) in a computer

game is no longer relevant with the narrative concept

of Trapped that mostly comprehended in the auditory

context. The notion to redefining the perceptual

appearance of the enemy object is required, in order

to initiate the perception-shifting from visual to the

auditory.

Therefore, the game enemy will no longer emerge

as the common antagonistic character (visual), but

instead through an embodiment of a sonic object that

emitting sounds to stimulate the episodic memory of

the player. This memory refers to collections of past

personal experiences that occurred at a particular time

and place which allow an individual to figuratively

travel back in time to remember the event that took

place at that particular time and place (Schacter,

Gilbert, Wegner, 2009, p. 185–6). However, due to

differences in hearing ability of each an individual,

we decided to not transforms the enemy appearance

‘Trapped’: An Interactive Gamiﬁed Audiovisual Installation

at once, but rather gradually in three phases according

to the three game-stages.

3.4 Environmental Design

In order to actualize the sonic unification concept, the

environmental design is splitting into the three main

stages that based on the game narrative and the player

hearing capability.

These stage divisions are comprehending as a

transition phase from the visual space perception of

our world into the Trapped' space perception that

dominated by auditory perception. The visual design

kept to a minimum but still engaging for the player,

below is the breakdown for each stage:

Figure 1. The three game-stages.

3.4.1 First Stage

Stage 1 (Green coloured) concept is to introduce how

the game works through basic gameplay, therefore,

the jumping features are still disabled in order to

simplify interactivity control. The scene has both

narrow and comprehensive platform, including a

ramp for the player to traverse. The Objective in this

stage is to find several enemies that generate sound

by listening to its sound direction; the enemy in this

level is visible and easy to find as the transition phase

for the player as mentioned previously.

Figure 2. First stage.

3.4.2 Second Stage

Once all enemies in the stage found, the player then

moves to Stage 2 (Orange). In this stage, the player

introduced to a maze-like interior. This type of

environment challenge player in memorizing rooms

and sound sources. The enemies in this stage have

little visual aid while still generating sound. There are

two areas connected with the bridge in this stage. The

first area consists of small rooms and a bit of window

for the player to re-orient their heading. The second

area is a bit wider but still feels claustrophobic. Player

need to listen carefully as the enemy might be on the

other side of the room.

Figure 3. Second stage.

3.4.3 Third Stage

Connected by a small bridge, the last stage is a

combination of rooms and open area from the first

stage. The researcher toned down the maze-like area

from the stage 2 and reintroduce a larger, this time

with verticality for the player to explore. The enemies

in this stage is now invisible and player needs to rely

on their hearing in order to find them. The reward for

finishing this stage is where the player raised from the

last platform, uncovering the whole stage for them to

see.

Figure 4. Third stage.

CREATIVEARTS 2019 - 1st International Conference on Intermedia Arts and Creative

3.5 Audio Components

The audio component is collections of sound that

employed to actualize the change of perception on the

enemy objects and interactive process from visual to

auditory perception. This component is categorized

into three different segments according to their

functionality as follows:

3.5.1 Enemy Sound

This component consists of fifteen different sounds

that will be emitting through fifteen different enemy

objects. The sound materials are created using the

results of field recording and the audio-mixing

process and categorized into two types of moods

based on the eight basic of human emotion proposed

by (Plutchik, 1997): 1) Joy—feeling happy,

happiness, gladness, and, 2) Sadness— feeling sad,

sorrow, grief. Next, we sorted out the human-related

activities or events that might be relevant or imply to

those moods, to define the kind of sound to use, as

follows:

• Joy: laugh, birthday party, clapping, children

playing, singing, concert ambiance, and game

arena.

• Sadness: cry, sobbing, war, demonstration,

riots, scream, yelling, and children' cry.

This component is a direct implication of the sonic

pathfinding and perception shifting, in which the

enemy is no longer presented in the visual domain.

3.5.2 Ambience

This component consists of the static sounds that

distributed evenly into the three-game stages — the

sound material created using the field recording

results without any mixing process. There is no

substantial interactivity role in its functionality,

beside implies the illusion of reality for the player.

3.5.3 User Input Feedback

The third audio component perhaps is the most

relevant with the audience’s interaction in the game

world. It does not "really" belong to the narrative

concept but is necessary inclusion, due to its unique

function for the game mechanic. Schütze (2013)

states that experience has found that omission of these

audio feedback cues can lead to confusion or

discomfort on the part of the player. In the general

computer game, user feedback sounds are usually the

beeps and clicks that acknowledge the player’s input

actions.

Trapped only use a single sound for this

component, i.e., the foot-step that corresponds to the

player's movement in 3D Gameworld. This feedback

component is like less required in conventional

interactive installation (non-Gamified), but they are

mandatory in gamified audiovisual installation.

3.6 Sonic Environment

3.6.1 Sound Formats

In general production of gamified audiovisual or

interactive audio works, there are two standard audio

formats that commonly in use. Two-dimensional (2D)

sounds are sounds that are consistently produced

through the speakers and have a specific level and

phase relationship, set by predetermined data. The

typical example of the 2D sound is audio CD and

Cassette as well as any regular media such as radio,

television or film. Three-dimensional (3D) sounds in

gamified context are sounds that placed in 3D

Gameworld that have volume attenuation based on

the distance from the listener, in which a 3D audio

engine governs the input and spatial parameters

(distance, direction, elevation, etc.).

Based on the environmental design that conceived

for the 3D environment, Trapped implement the high-

quality binaural rendering (third order-HRTF) to

render the audio components and adopting the first-

person perspective techniques to determine the player

position relevant to spatial location. This technique

places the game camera in the position of the player

as if they were in the world itself; the player looks

through the camera as if they were looking at the

world through their own eyes. All things in the world

will be relative to this perspective, and as such, the

audio must sound as it would if the player was

actually in the game world (Schütze, 2013).

3.6.2 Sound Engine

In the computer game, the audio engine is a

component that consists of algorithms related to the

loading, modifying, and output of sound through the

speaker system in both 2D or 3D format.

Google resonance audio (GRA) is a multiplatform

3D audio SDK that able to produces a realistic

experience for AR, VR, gaming, and video. It’s also

capable of replicating the similar characteristic of the

sonic environment like they occurred in the physical

‘Trapped’: An Interactive Gamiﬁed Audiovisual Installation

world as long as acoustic measurement data is

available.

GRA will automatically calculates the spatial

parameters such as Doppler, echo, and reverb

according to their spatial location within the 3D

Gameworld. GRA’s input should be in monophonic,

due to audio spatial positions is determined by the

spatial camera location in the game world, i.e.,

relative to the current camera view and active

perspective of the player.

The 3D sound engine only implemented on the game

enemy audio component to recreate the immersive

sonic environment of the real world so that the player

could sense the enemy sonically as it is on the

physical world. Though enemy objects are placed in

random, they are passively stationer at the given

coordinate and waiting to be found by the player.

Thus, the spatial parameter and binaural

experience are determined by the player position that

corresponds to the game camera. At the time of

development, GRA is the most practical solution for

sound spatialisation because it can be used directly

inside the unity3D and powerful enough to actualize

the spatial idea of Trapped.

3.6.3 Sonic Interaction

The latest generation of computer game technology

has offered a new form of interactivity that never been

achieved previously. Interactivity is an essential

element in the computer game that presents an

embodiment of the communication between the

player and the virtual environment.

Lievrouw & Livingstone (2002) states,

Interactivity can be defined into three interactivity

form of an emerging communication system: user-to-

user interactivity, user-to-documents interactivity,

and user- to-system interactivity.

Following the general design of trapped, that aims

to implement auditory perception at the centre of

interactivity process. Trapped interactivity can be

defined into two: 1) player-to-enemy interactivity

(hearing) and, 2) player-to-environment interactivity.

A noise cancelling headphone is used to listen to the

enemy sound, whereas the 360 Xbox controller

employed to interact with the game environment

through a mapping value from the player's movement

as follows: 1) left thumbstick—directions, 2) right

thumbstick— camera rotation and, 3) left trigger—

jump. These interactivities occur on the three-game

stages and based on the narrative concept of

environmental design, as explained previously. The

term of Gamified Sonic Interaction (GSI) is used to

define the player's interactivity on Trapped. The word

of gamified is derived from Gamification concept

which according (Robson et al., 2015) defined as a set

of activities and processes to solve problems by using

or applying the characteristics of game elements.

Therefore, GSI is a multisensory interaction based on

game design elements, where in this context is the

Trapped’ narrative concept.

4 CONCLUSIONS

This paper discusses the concept and technical

approaches behind the creation of "Trapped" that

focuses on the sonic interactivity between the player

and the game-enemy. Trapped offers not only the

immersive audiovisual but also storytelling narrative

that constructs multisensory interaction that

manifests perceptual, cognitive, and psychological

experiences within the virtual 3D environment.

REFERENCES

Ciciliani, M., & Lüneburg, B. (2018). Gamified

Audiovisual Works – Composition, Perception,

Performance. Emille, the Journal of the Korean

Electro-Acoustic Music Society.

Climent, R., (2015), 'Putney' for game-audio game-audio

interactive composition [online]. Available at:

https://bit.ly/2EXbaku

eContact! 17.4 — Analogue and Modular Synthesis:

Resurgence and Evolution (February / février 2016).

Montréal: Communauté électroacoustique canadienne /

Canadian Electroacoustic Community.

Hickson, F., & Newton, V. (1981). Sound localization: Part

I: A brief history. The Journal of Laryngology &

Otology, 95(1), 29-40.

doi:10.1017/S002221510009037X

Lievrouw, L. and Livingstone, S. (Eds.). (2002). Handbook

of New Media: Social Shaping and Social

Consequences. London: Sage.

Plutchik, Robert (1997-01-01). The circumplex as a general

model of the structure of emotions and personality.

American Psychological Association. pp. 17–45.

Robson, K., Plangger, K., Kietzmann, J., McCarthy, I. &

Pitt, L. (2015). "Is it all a game? Understanding the

principles of gamification". Business Horizons.

Robson, K., Plangger, K., Kietzmann, J., McCarthy, I. &

Pitt, L. (2015). "Is it all a game? Understanding the

principles of gamification". Business Horizons.

CREATIVEARTS 2019 - 1st International Conference on Intermedia Arts and Creative

Schacter DL, Gilbert DT, Wegner DM (2009). "Semantic

and episodic memory." Psychology. pp. 185–6.

SCHÜTZE, S. (2003). The creation of an audio

environment as part of a computer game world: The

design for Jurassic Park – Operation Genesis on the

XBOX™ as a broad concept for surround installation

creation. Organised Sound, 8(2), 171-180.

doi:10.1017/S1355771803000074.

Simons, Jan., (2004), Narrative, Games, and Theory, the

international journal of computer game research,

volume 7, (issue 12007 ISSN:1604-7982) [online].

Available at: https://bit.ly/31fVMcA

‘Trapped’: An Interactive Gamiﬁed Audiovisual Installation