Assessing the Usability of Different Virtual Reality Systems for

Fireﬁghter Training

Fabrizio Corelli, Edoardo Battegazzorre, Francesco Strada, Andrea Bottino and Gian Paolo Cimellaro

Department of Control and Computer Engineering, Politecnico di Torino, Corso Duca degli Abruzzi 24, 10129 Torino, Italy

Keywords:

VR Training, Serious Games, Locomotion, Usability, User Study.

Abstract:

The use of Virtual Reality (VR) based learning environments for training ﬁreﬁghters is becoming more and

more common. The key advantages of these approaches is that they allow the development of experiential

learning environments, where trainees can be involved into and interact with complex emergency scenarios,

including those that cannot rely for the training on real world systems and environments due to costs or

security concerns. Despite that, current VR training systems are still affected by a number of weaknesses,

mainly related to usability and to the (limited) sense of presence conveyed by the virtual environment (VE),

which can negatively affect the expected learning outcomes. To this end, in order to gain further insight into

this problem, this work aims at assessing the usability of a ﬁreﬁghter training application deployed in three VR

systems and exploiting serious games in the educational approach. The VR systems under analysis provide

different levels of immersion and offer different approaches to manage interaction and locomotion inside

the VE. Experimental results, obtained through a user study, show differences among the three systems. In

particular, the devices and metaphors used to manage locomotion in VR seem to be the most critical parameters

with respect to usability and learners’ achievements.

1 INTRODUCTION

Current VR technologies are providing ﬁre services

with new and compelling opportunities for transform-

ing the way ﬁreﬁghters are trained. According to (En-

gelbrecht et al., 2019), the key advantages of VR-

based ﬁreﬁghter training can be summarized as fol-

lows. First, ﬁreﬁghters need to learn how to address

critical situations where they can risk their life, and

VR provides a safe training environment where emer-

gencies can be simulated without putting trainees in

any real danger. Second, a VR application allows

trainees to experience different scenarios and emer-

gency procedures and the same application can be

used to train different categories of personnel. Third,

VR training can be also delivered remotely, thus help-

ing reduce costs and overall training times, and it

can even support collaborative activities, which are

beneﬁcial for learning (Rojas-Drummond and Mer-

cer, 2003). Then, VR offers compelling experien-

tial learning environments that improve the learner’s

engagement and, ultimately, the learning outcomes.

Another advantage is that VR promotes repetition

(i.e., users can repeat the training session at their own

pace), which, in turns, improves content retention. Fi-

nally, training sessions can be logged and reviewed

in a debrieﬁng sessions, where learners can critically

analyze what they did, get insights from their experi-

ence and analyze mistakes and explore alternative so-

lutions. Thus, debrieﬁng sessions enhance the trans-

fer of knowledge and skills from virtual to real world,

and promote active learning (Garris et al., 2002).

The idea of using VR for ﬁreﬁghter training is

not a new concept. Several works have been pre-

sented in the literature. The VR systems used to

support the training sessions can vary from desktop

VR (Lee et al., 2010) to immersive systems, such

as those based on Head Mounted Displays (HMDs),

(Argasinski et al., 2018), (Tate et al., 1997) or on

CAVE environments (Backlund et al., 2007). Simu-

lations can also exploit dynamic and physics based

systems for recreating the correct behaviour of ﬁre

and smoke (Cha et al., 2012). This is a crucial el-

ement in this context, especially when training tar-

gets emergency procedures inside buildings or tun-

nels. Some attempts have been even made to increase

the sensory spectrum of these simulations by imple-

menting odor (Cater, 1994) and heat radiance genera-

tors (Lee et al., 2010) to improve the sense of immer-

sion into a ﬁery environment. Given the relevance of

146

Corelli, F., Battegazzorre, E., Strada, F., Bottino, A. and Cimellaro, G.

Assessing the Usability of Different Virtual Reality Systems for Fireﬁghter Training.

DOI: 10.5220/0008962401460153

In Proceedings of the 15th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2020) - Volume 2: HUCAPP, pages

146-153

ISBN: 978-989-758-402-2; ISSN: 2184-4321

the decision-making aspect in the emergency manage-

ment, VR environment have been used as well to an-

alyze the relationships among ﬁreﬁghter experience

and the decision-making processes (Bayouth et al.,

2013).

Despite the many advantages, there are a number

of weaknesses that still affect current VR systems in

this speciﬁc context (Engelbrecht et al., 2019). One

relevant issue is that, despite the many efforts, actu-

ally there are no devices or interaction metaphors that

allow natural movements in large VR spaces, such as

those that are typical in ﬁreﬁghter training scenarios.

Moreover, some of these navigation approaches might

even induce cybersickness (Rebenitsch and Owen,

2016), with a negative effect on learning. A second

weakness is that technology still needs a leap forward

to maximize the sense of presence, which ultimately

is one of the main factors that help transfer the learned

skills from virtual space to real life. Finally, since

trainees’ acceptance of the technology is essential to

achieve the desired learning outcomes (Heldal et al.,

2016), the usability of the system and the overall user

experience should be maximized as well. However,

achieving this result is not trivial and involves a care-

ful selection of the interaction devices/metaphors and

an ad-hoc design of the application.

The goal of this work is to assess the usability of

three different VR systems (using different hardware

conﬁgurations) in a ﬁreﬁghter training context. In de-

tails, these systems are a desktop VR, and two im-

mersive environments, both leveraging an HMD for

visualization and hand-held controllers for interaction

but exploiting two different locomotion techniques:

a gaze-directed-steering metaphor (Bowman et al.,

1997) and an active re-positioning technique (Nils-

son et al., 2018) that allows trainees to naturally walk

through large VEs leveraging an omni-directional

treadmill. The training scenario used in the exper-

iments reproduces a ﬁre-following-earthquake event

in a school. The educational path of the application

comprises a learning mode (where trainees are in-

structed step-by-step on the sequence of actions they

have to perform in the emergency procedure), and an

evaluation mode (which leverages a serious game to

assess the learned skills).

The three VR system have been compared by

means of an user-evaluation study that involved 45

volunteers divided in three separate groups (each us-

ing a different system). Experimental results showed

differences in the usability of the three system un-

der analysis, highlighting that the way locomotion is

managed is the most critical parameter that affects

both usability and users’ achievements in terms of

learning outcomes.

The rest of the paper is organized as follows. Sec-

tion 2 details the design of the training application and

the characteristics of the different VR systems under

analysis. Section 3 introduces the experimental pro-

tocol and Section 4 presents and discusses the results.

Finally, Section 5 draws the conclusions of this work.

2 APPLICATION AND SYSTEM

DESIGN

The ﬁreﬁghter training programs, irrespective of the

actual simulation scenario addressed, require to carry

out certain actions in a speciﬁc order. The comple-

tion of an action involves navigation in the VE and

interaction with virtual places, objects and avatars in-

side the environment. That said, the application de-

sign should be ﬂexible enough to support different

scenarios and procedures. To this end, we modeled

action dependencies as a directed graph, where nodes

represent individual actions and edges correspond to

dependency requirements. The control of the node

execution ﬂow leverage context awareness (i.e., envi-

ronment state, user’s interaction, internal and external

events). Composite nodes can be used to orchestrate

various sub-nodes (according to different algorithms,

such as sequential or parallel execution and loop man-

agement), thus allowing the deﬁnition of complex ac-

tion inter-dependencies.

As another constraint, the design should sup-

port the application deployment on different hardware

conﬁgurations. This has been done by leveraging de-

sign patterns speciﬁc for multi-platform applications

and exploiting the cross-platform features offered by

the Unity engine for the development.

2.1 Learning Path Design

The learning path of the application is organized in

two different parts: a learning and an evaluation ses-

sion. In the learning session, users are guided step-

by-step through the correct sequence of actions they

have to perform. For each individual action, visual

and audio clues instruct trainees on what they have to

do (and why) and on the sequence of steps required to

complete the activity. During task accomplishment,

users are supported by prompt and clear feedback that

inform them when an interaction is available (such as

adding a glow to interactable objects) and notify the

success/failure in performing an action.

The evaluation session is envisioned as a seri-

ous game, where users can freely perform any ac-

tion involved in the procedures learned in the previ-

ous phase, but they cannot beneﬁt from any of the

Assessing the Usability of Different Virtual Reality Systems for Fireﬁghter Training

147

cognitive aids available in the learning mode. The

game design leverages two main elements. A timer

enforces a time limit for completing a given task.

A score value translates the player’s success in the

game into a numerical representation, which is then

used as an automatic assessment of the learned skills.

The current score and timer values are displayed in

the VE and audio and visual cues highlight speciﬁc

events associated to them (e.g., timer expiration, new

points obtained). At the end of the game, players re-

ceive a summary report of their achievements and can

see their placement in the overall ranking. This fea-

ture aims at fostering competitive behaviors (i.e., by

challenging players to beat their colleagues’ scores),

which ultimately are beneﬁcial to learning (Cagiltay

et al., 2015).

2.2 VR Environments

For the evaluation, the application was deployed in

the three types of VR environments depicted in Fig-

ure 1, whose main differences consist in the interac-

tion techniques and the level of immersion offered.

The ﬁrst is a desktop VR system (referred to as

DVR in the following), where the VE is displayed

on a large monitor. The mouse movements control

camera orientation, while the player’s spatial move-

ments are handled with the keyboard arrow keys. The

behaviour associated to an interactable object can be

triggered by ﬁrst selecting the object (pointing the

camera at it) and then pressing a mouse button.

On the contrary, the immersive VR system (IC)

uses an HMD for the visualization and hand-held con-

trollers for managing interaction and locomotion. The

head movements (captured by the HMD tracking sys-

tem) are used to control camera position and orien-

tation. As for the navigation, the active area of the

HMD trackers is not large enough to let user explore

the VR by physical walking. Therefore, locomo-

tion is managed by using the joystick included within

the controllers to translate the user in the current

gaze yaw direction. Interaction with objects leverages

the virtual hand metaphor (Poupyrev and Ichikawa,

1999), where the controller movements are mapped

into that of two virtual hands that players can use to

grab or activate interactable objects in the VE, thus

providing a natural and immediate interaction tool.

The last system (KAT) is similar to IC, since it

leverages HMD as display system and hand-held con-

trollers for interaction with objects, but uses a KAT-

Walk treadmill to manage locomotion. Treadmills are

“body-centric” re-positioning systems (Nilsson et al.,

2018) that translate physical gestures (e.g. walking

or running) into virtual movements. With the KAT-

Walk, the user is strapped into a harness, attached to

a supporting structure and slightly lifted over a con-

cave platform (Figure 1). The platform has a low fric-

tion surface that prevents the forces generated during

each step from physically moving forward the user.

The walking gestures are captured by two inertial sen-

sors that are placed on special overshoes, while a third

sensor, placed on the back of the harness, tracks the

movement direction and triggers the locomotion input

when the user tilts forward or backward.

Figure 1: VR environments considered in this study: desk-

top VR (DVR, left), immersive VR (IC, center) and tread-

mill based immersive VR (KAT, right).

3 EXPERIMENTS

The scenario selected for our experiments is a ﬁre-

following-earthquake event in a school. In order to

make it as realistic as possible, the emergency sce-

nario is set into a real building (the Mascagni mid-

dle school of Melzo, Italy), which was recreated in

the VE importing its Building Information Model

(BIM). In the emergency scenario under analysis, a

short circuit caused by an earthquake generates a ﬁre

inside a classroom, located in the ﬁrst ﬂoor of the

school. Some students in the classroom are wounded

or trapped by fallen debris and, thus, they need to be

assisted and rescued, while the remaining students es-

cape from the classroom and evacuate through the ex-

ternal emergency stairs. The activities of the emer-

gency procedure trainees are supposed to learn, as

well as their completion times, have been deﬁned with

the help of expert ﬁreﬁghter trainers according to the

characteristics of the building where the procedure

takes place. These activities are the following. The

ﬁreﬁghter should enter the main building, identify

the ﬁre location by analyzing the smoke propagation

and the screams coming from the classroom. Since

the operator is likely to enter the school for the ﬁrst

time, he/she must observe an evacuation plan chart

hanging on the wall in order to orient himself/herself

into the unfamiliar environment. Then, he/she must

ﬁnd his/her way to the ﬁrst ﬂoor (navigating corri-

HUCAPP 2020 - 4th International Conference on Human Computer Interaction Theory and Applications

148

Figure 2: Snapshot of the VR training scenario (from left to right): the main entrance of the school, the computer room where

the ﬁre started, the evacuation meeting point seen from the emergency stairs and the ﬁre-extinguishing step.

dors and using internal stairs). When the operator has

reached the classroom, he/she should ﬁrst indicate to

the unharmed students the closest emergency exit so

that they can promptly evacuate the building. Then,

he/she must provide ﬁrst aid to the injured students

and help them evacuate to a safe area. Subsequently,

the learner has to go back in the classroom and op-

erate a ﬁre extinguisher (available in the building).

Once the ﬁre is extinguished, the operator should re-

turn to the safe area where the evacuated students are

waiting for him. Some snapshot of the procedure ac-

tions can be seen in Figure 2. We underline that the

scenario taken in consideration allows us to stress the

interaction in general and the locomotion interfaces

in particular, since users had to (virtually) walk for

long distances and move inside narrow spaces (thus

requiring a ﬁne-grained and detailed control of their

movements) in order to complete the assigned tasks.

In order to compare the different setups described

in Section 2.2, we performed a user study that in-

volved 45 volunteers (35 men and 10 women, aged

between 29 and 30) selected among students and

personnel of our University. Six of them are fre-

quent users of VR, 24 have had a previous experience

with VR and the remaining 15 had never experienced

VR before. Users were divided in three groups of

15 people, each experiencing the training application

through a different system. The experimental proto-

col applied was the following. First, since users are

likely to have low familiarity with the VR devices to

be used, the learning experience started with an in-

teraction training session in a test environment where

users could get acquainted with the interaction and lo-

comotion modalities available. This preliminary step

aimed at reducing barriers related to VR technologies

(especially for novices). When users felt conﬁdent

with the system, they were invited to repeat twice

the learning session and, ﬁnally, to perform twice the

evaluation session.

The assessment of the different systems encom-

passed the analysis of both the learning outcomes and

the usability of the system. In the human-computer

interaction (HCI) ﬁeld, usability is deﬁned as “the

extent to which a product can be used by speciﬁed

users to achieve speciﬁed goals with effectiveness, ef-

ﬁciency, and satisfaction in a speciﬁed context of use”

(Iso, 1998). Following this deﬁnition, we assessed

usability by measuring effectiveness in terms of cor-

rectness of the performed procedures, efﬁciency from

time on task and paths navigated inside the environ-

ment (where both effectiveness and efﬁciency met-

rics using objective in-game analytics), and satisfac-

tion through standardized questionnaires (i.e., lever-

aging subjective measurements) that users were re-

quired to ﬁll after they completed the experience. The

proposed questionnaire is organized in different sec-

tions aimed to collect (i) information about usability

in broad terms (by considering the System Usability

Scale, SUS (Brooke, 2013), and the ﬁve attributes de-

ﬁned by Nielsen (Nielsen, 1993), i.e., learnability, ef-

ﬁciency, memorability, possibility to recover from er-

rors and satisfaction), (ii) ergonomic aspects of the

proposed system (through statements derived from

the ISO 9241-400 standard (iso, 2007)) and (iii) de-

tailed usability information on the VR system (on the

basis of the VRUSE questionnaire (Kalawsky, 1999)).

All questionnaire items had to be scored by users ex-

pressing their agreement on a ﬁve point Lickert scale

(1, totally disagree; 5, totally agree).

As for the expected learning outcomes, since

trainees should learn how to execute the various steps

of the procedure in the correct order and within a pre-

deﬁned time interval, their assessment was based on

the analysis of users’ behaviour in terms of comple-

tion time, correctness of the performed procedures

and learning progresses.

4 RESULTS AND DISCUSSION

In this section, we will ﬁrst discuss the results con-

cerning usability of the VR systems under analysis

(Section 4.1. Then, we will assess the learning out-

comes achieved with these systems and with the help

of the serious game used during the evaluation phase.

In the following, both subjective and objective re-

sults obtained from the three volunteers groups were

analyzed with a One-Way ANOVA test with post-hoc

Tukey’s test to eventually identify groups showing

statistically signiﬁcant differences.

Assessing the Usability of Different Virtual Reality Systems for Fireﬁghter Training

149

4.1 Usability Assessment

A ﬁrst indication of the usability of the three environ-

ments comes from the SUS normalized results in the

0–100 range (with odd items reverted). According to

(Brooke, 2013), a score above 68 shall be considered

as above average. As shown in Fig. 4, the KAT SUS

score is signiﬁcantly lower than that of the other two

systems (DVR 86.0, IC 81.7, KAT 68.3, p = 0.00002)

and, thus, its usability can be considered as barely ac-

ceptable. Similar conclusions can be drawn by con-

sidering Nielsen’s attributes (second section of the

questionnaire). The three systems did not show any

signiﬁcant difference in the learnability, memorabil-

ity and satisfaction attributes. However (Figure 3),

the KAT had signiﬁcantly lower values than DVR and

IC (DVR 4.7, IC 4.5, KAT 4.1, p = 0.001) regard-

ing efﬁciency and was also perceived as having lower

possibilities to recover from errors than IC (DVR 3.9,

IC 4.4, KAT 3.7, p = 0.03).

These efﬁciency results are conﬁrmed by the ob-

jective results collected. As a matter of fact, if we

consider the average distances travelled with the three

systems in the ﬁrst and second run of the evaluation

session (Figure 6), we can observe a large difference

between the KAT and the other systems (DVR 262.2,

IC 241.5, KAT 284.4, p = 0.01 for the ﬁrst run; DVR

254.8, IC 213.6, KAT 277.2, p = 0.001 for the second

run, where all measurements are expressed in meters).

Similar differences between KAT and DVR/IC are

obtained from the analysis of the percentage of ac-

tions completed in time (ﬁrst run: DVR 95.0%, IC

90.08%, KAT 75.8%, p = 0.02; second run: DVR

97.5%, IC 96.4%, KAT 83.0%, p = 0.01; see Fig-

ure 6). This result highlights as well the negative ef-

fect of the lower KAT usability on the learning out-

comes. Trainees should not only learn to perform all

the actions required (and in the correct order), but they

should also learn to complete them in a suitable time,

since (for some actions) time is a critical parameter

(e.g., in our case, the room where ﬁre started should

be reached as soon as possible in order to provide im-

mediate ﬁrst-aid to the people in that place).

A possible explanation of this last result is pro-

vided by both direct observations and analysis of the

path traveled. Observations highlighted that KAT

users were less capable than others to exert a ﬁne-

grained control of their movements. As a result, they

were not always in the condition of following an “op-

timal” path to reach their destination. These difﬁcul-

ties in controlling small and detailed movements in

narrow spaces

are probably the main reason that led

In our simulation, this is, for instance, the situation

trainees are faced with when they have to operate in the

to increase the completion times of individual actions

and, thus, of the full procedure.

Another element that contributes to highlight the

navigation issues with KAT is the analysis of the trav-

elled paths, which are summarized in the heatmaps

shown in Figure 5. In each heatmap, colors on the

ﬂoor correspond to different trafﬁc intensities (red

areas are the ones lots of people walked in, while

green regions had lower trafﬁc). It can be seen that

user tracks for DVR and IC are much more compact

than the KAT’s one (despite few outliers that took

“bizarre” paths to reach their targets).

As for ergonomics (third section of the question-

naire), learners were asked to evaluate the interaction

with the VR systems according to the four statements

summarized in Figure 3. It can be seen that three out

of these four statements underline the low KAT er-

gonomics. In details, the locomotion device is more

cumbersome than the ones provided by DVR and IC

(DVR 1.4, IC 1.6, KAT 2.4, p = 0.005) and, to be op-

erated, it requires a higher mental effort than DVR

(DVR 1.7, IC 1.8, KAT 2.5, p = 0.03) and higher

physical efforts than both DVR and IC (DVR 1.1, IC

1.5, KAT 3.8, p = 5.53e − 13).

The fourth questionnaire section (VRUSE) under-

lines again the lower user appreciation received by

KAT. The VRUSE is divided into different subsec-

tions (related to the analysis of functionality, loco-

motion, interaction with objects, ﬂexibility, error cor-

rection/handling, simulation ﬁdelity, and sense of im-

mersion and presence). Each subsection is then con-

cluded by a closing summary question aimed at cap-

turing from users an overall evaluation on the aspects

investigated by the subsection. If we start analyzing

these closing questions, it can be observed that the an-

swers are similar for all the systems

, exception made

for the question corresponding to the locomotion sec-

tion, where the KAT had signiﬁcant lower values than

DVR and IC (DVR 4.1, IC 3.9, KAT 3.2, p = 0.04).

Thus, in order to gain further understanding about

the issues encountered by users, we deemed interest-

ing to discuss in details the questions of this subsec-

tion and their answers (which are summarized, again,

in Figure 3). A ﬁrst negative comment is that the KAT

classroom where the ﬁre started, or when they have to climb

staircases.

As a note, the fact that the level of immersion and pres-

ence delivered by DVR was similar to the one offered by

IC and KAT was quite a surprising ﬁnding. A possible ex-

planation is that each user group experienced a single VR

system (thus, volunteers had no possibilities to make com-

parisons among them) and DVR group was experiencing

the VE through large monitors, which probably helped mit-

igate the “through the window” effect inherent in desktop

VR systems.

HUCAPP 2020 - 4th International Conference on Human Computer Interaction Theory and Applications

150

Figure 3: Excerpt of the subjective evaluation results. Top row: overall scores for the most signiﬁcant Nielsen attributes (Left)

and the ISO 9241-400 properties (Right). Bottom row: usability factors regarding the locomotion system for which we found

a signiﬁcant difference among systems (questions adopted from the VRUSE questionnaire). For all graphs, the “*” symbol

indicates a statistically signiﬁcant difference and standard deviations are expressed through error bars.

Figure 4: SUS summary results. The scales displayed are

taken from (Brooke, 2013) and show that while IC and DVR

score as good/excellent, KAT can be only considered as

marginally acceptable.

ease of use is sensibly lower than that of the other two

locomotion approaches (DVR 4.0, IC 3.9, KAT 2.8,

p = 0.003). Both locomotion systems in immersive

environments (IC and KAT) are not highly appreci-

ated by their users, since they expressed their pref-

erence to use a different locomotion system with a

sensibly higher strength than that used by DVR users

(DVR 1.6, IC 3.2, KAT 2.7, p = 0.002). Then, learn-

ing how to use the KATWalk appears to be more dif-

ﬁcult than the mouse and keyboard controls offered

by DVR (DVR 1.3, IC 1.7, KAT 2, p = 0.04) and

the KAT sensitivity is higher than that of IC (DVR

2.3, IC 1.7, KAT 2.8, p = 0.02). Finally, the proba-

bility of committing errors with KAT is higher than

with DVR (DVR 1.6, IC 2.3, KAT 2.7, p = 0.04) and

the level of control is lower with KAT than with DVR

(DVR 4.0, IC 3.4, KAT 2.8, p = 0.005). These level

of control values conﬁrm the difﬁculties experienced

by KAT users in the (ﬁne) control of the navigation.

4.2 Learning Outcomes

As we already observed, the VR application was ef-

fective in producing the expected learning outcomes.

The percentage of actions completed in the correct

order in the second evaluation run was 100% for all

users except one volunteer of the DVR group who

made a single mistake. The percentage of actions

completed in the correct order and in a timely fash-

ion (Figure 6, second run) was close to 100% for all

groups, except for KAT where this value was only

83% (and signiﬁcantly different from that of the other

groups). This result, as we discussed before, was

mainly due to the issues with the locomotion device.

Assessing the Usability of Different Virtual Reality Systems for Fireﬁghter Training

151

Figure 5: Heatmaps indicating users’ movements in the virtual environment. Images are color coded (green: less frequently

travelled paths, red: more frequent ones) and are divided for group (from left to right: DVR, IC and KAT) and ﬂoor (for each

group, top image shows the ground ﬂoor and bottom image the ﬁrst ﬂoor).

Figure 6: Total distances traveled (up) and percentage of

activities completed in time (down) for the two runs of the

evaluation session.

Figure 6 allows to appreciate as well that, despite

small inter-session differences, all volunteer groups

beneﬁts from repetition learning. The percentage of

correct action completion increased between the two

runs by 1.7% for both DVR and IC and 8.3% for KAT,

and that of correct and timely completion of 2.5% for

DVR, 5.6% for IC and 7.2% for KAT. Despite that, the

lack of follow-up studies does not allow to infer the

effects of repetition on users’ knowledge retention.

Finally, we can analyze the contribution provided

to the learning outcomes by game mechanics intro-

Table 1: Statements used to assess users’ appreciation of the

serious game (mean values and standard deviations).

Statement Score (SD)

I had fun using a game to learn intended content 4.56 (0.58)

I would have preferred an instructor-based learning

modality

2.22 (0.94)

I would have preferred a learning modality, based on

books, notes, slides, etc.

1.38 (0.74)

The designed game is a valid learning tool 4.42 (0.61)

The possibility to compare my score with other learn-

ers’ scores made me try to improve my results

4.40 (1.04)

I found the tool more a game than a system suitable for

training

2.78 (1.09)

Trying to improve my results in the game let me learn

intended content better

4.02 (1.02)

The presence of a timer stimulated me to quickly carry

out required actions

4.07 (1.06)

I found the timer a stressful element 1.78 (0.84)

duced in the evaluation session. This analysis is based

on the concluding section of the questionnaire, in

which we asked volunteers to express their level of

agreement with the 9 statements reported in Table 1.

We found that the results have no signiﬁcant differ-

ences among the three groups and, thus, we reported

them as the cumulative averages among all users of

the three groups. These results can be summarized as

follows. The game features made the learning process

more fun and the approach proposed to deliver the

educational contents was appreciated by users. The

game elements introduced to solicit the competitive-

ness (the score and the leaderboard) were able to im-

prove the learners’ engagement and foster repetition,

and the timer was considered as an effective element

in encouraging users to complete the procedure ac-

tions in time. Finally, volunteers found that there

was a good balance between educational and enter-

tainment elements.

HUCAPP 2020 - 4th International Conference on Human Computer Interaction Theory and Applications

152

5 CONCLUSIONS

In this work we presented and discussed a user study

aimed at assessing the usability, ergonomics and ef-

fectiveness (in terms of learning outcomes) of three

different VR systems used to deploy a ﬁreﬁghter

training application.

The major takeaways of this work are the follow-

ing. First, the VR training environment was capable

of achieving the expected learning outcomes, in terms

of both precision and timely completion of the emer-

gency procedures. Second, the introduction of the

serious game in the evaluation phase of the applica-

tion was appreciated by learners and was contributing

to support the educational path designed. Third, we

found different levels of usability (as well as different

levels of users’ appreciation) among the different VR

systems analyzed.

In particular, one of the most critical parameters

inﬂuencing the evaluations was the quality of the lo-

comotion management provided by the system. As

a matter of fact, locomotion is a relevant task in our

scenario. In order to complete the emergency proce-

dures, users have both to travel long distances and to

exert a ﬁne control on their movements. With respect

to these requirements, when the interaction devices

(or the interaction metaphors) are not capable of sup-

porting the users, the results is a negative effect on

the trainee performances, in terms of both accuracy

and timing of the execution.

Future works will address the evaluation of alter-

native locomotion interfaces, simple to use and capa-

ble of guaranteeing a high level of immersion and an

adequate naturalness in the locomotion control. Then,

given the relevance of locomotion in other areas of ap-

plication, we are planning to extend the breadth of this

study beyond the ﬁreﬁghter training domain.

REFERENCES

(2007). ISO 9241-400 Ergonomics of human-system inter-

action – Part 400: Principles and requirements for

physical input devices.

Argasinski, J. K., Wegrzyn, P., and Strojny, P. (2018). Af-

fective vr serious game for ﬁreﬁghter training. In

Workshop on Affective Computing and Context Aware-

ness in Ambient Intelligence, volume 41, page 43.

Backlund, P., Engstrom, H., Hammar, C., Johannesson, M.,

and Lebram, M. (2007). Sidh–a game based ﬁreﬁghter

training simulation. In 2007 11th International Con-

ference Information Visualization (IV’07), pages 899–

907. IEEE.

Bayouth, S. T., Keren, N., Franke, W. D., and Godby, K.

(2013). Examining ﬁreﬁghter decision-making: how

experience inﬂuences speed in process and choice. In-

ternational Fire Service Journal of Leadership and

Management, 7:51.

Bowman, D. A., Koller, D., and Hodges, L. F. (1997).

Travel in immersive virtual environments: an evalu-

ation of viewpoint motion control techniques. In Pro-

ceedings of IEEE 1997 Annual International Sympo-

sium on Virtual Reality, pages 45–52.

Brooke, J. (2013). Sus: A retrospective. Journal of Usabil-

ity Studies, 8(2):29–40.

Cagiltay, N. E., Ozcelik, E., and Ozcelik, N. S. (2015). The

effect of competition on learning in games. Computers

& Education, 87:35 – 41.

Cater, J. P. (1994). Smell/taste: odors in reality. In Pro-

ceedings of IEEE International Conference on Sys-

tems, Man and Cybernetics, volume 2, pages 1781–

vol. IEEE.

Cha, M., Han, S., Lee, J., and Choi, B. (2012). A virtual

reality based ﬁre training simulator integrated with ﬁre

dynamics data. Fire Safety Journal, 50:12–24.

Engelbrecht, H., Lindeman, R., and Hoermann, S. (2019).

A swot analysis of the ﬁeld of virtual reality for ﬁre-

ﬁghter training. Frontiers in Robotics and AI, 6:101.

Garris, R., Ahlers, R., and Driskell, J. E. (2002). Games,

motivation, and learning: A research and practice

model. Simulation & Gaming, 33(4):441–467.

Heldal, I., Wijkmark, H., and Pareto, L. (2016). Simulation

and serious games for ﬁreﬁghter training: Challenges

for effective use. NOKOBIT, 24:12.

Iso, W. (1998). 9241-11. ergonomic requirements for ofﬁce

work with visual display terminals (vdts). The inter-

national organization for standardization, 45.

Kalawsky, R. S. (1999). Vruse – A computerised diagnostic

tool for usability evaluation of virtual/synthetic envi-

ronment systems. Applied Ergonomics, 30(1):11–25.

Lee, J., Cha, M., Choi, B., and Kim, T. (2010). A

team-based ﬁreﬁghter training platform using the vir-

tual environment. In Proceedings of the 9th ACM

SIGGRAPH Conference on Virtual-Reality Contin-

uum and its Applications in Industry, pages 299–302.

ACM.

Nielsen, J. (1993). Usability Engineering. Academic Press.

Nilsson, N. C., Seraﬁn, S., Steinicke, F., and Nordahl, R.

(2018). Natural walking in virtual reality: A review.

Computers in Entertainment (CIE), 16(2):8.

Poupyrev, I. and Ichikawa, T. (1999). Manipulating objects

in virtual worlds: Categorization and empirical evalu-

ation of interaction techniques. Journal of Visual Lan-

guages & Computing, 10(1):19–35.

Rebenitsch, L. and Owen, C. (2016). Review on cybersick-

ness in applications and visual displays. Virtual Real-

ity, 20(2):101–125.

Rojas-Drummond, S. and Mercer, N. (2003). Scaffolding

the development of effective collaboration and learn-

ing. International journal of educational research,

39(1-2):99–111.

Tate, D. L., Sibert, L., and King, T. (1997). Using vir-

tual environments to train ﬁreﬁghters. IEEE Computer

Graphics and Applications, 17(6):23–29.

Assessing the Usability of Different Virtual Reality Systems for Fireﬁghter Training

153