Interaction Challenges in the Development of a Fire Warden VR
Training System using a HMD
Helen V. Diez
, Aitor Moreno
and Alejandro Garcia-Alonso
Vicomtech-IK4, San Sebastian, Spain
University of the Basque Country, San Sebastian, Spain
Virtual Reality, Head-Mounted Display, Interaction Devices.
This paper presents challenges encountered when interacting with 3D environments by means of Virtual Re-
ality devices in a seated position. A use case has been devised and developed to support this work: a fire
warden virtual training system. The users interact with the virtual environment with a combination of a Head-
Mounted Display and other assisting devices to complement the virtual experience (keyboard, gamepad, leap
motion). This work points out the benefits achieved using virtual reality in this environment and it exposes the
limitations experimented by humans in such situations. With the knowledge gathered with these experiments
this paper proposes interaction techniques to overcome them.
In the last years, the new generation of Head-Mounted
Displays (HMDs) have become popular devices to
provide highly immersive experiences which are very
difficult to achieve by other means. They are excellent
tools to view virtual environments (VEs) in a realistic
way but, by themselves, they do not offer interactive
capabilities with the objects in the VE. Additional de-
vices are attached to the HMD setup to overcome this
When wearing a HMD, users cannot see the real
world surrounding them, so it is difficult to locate
and use the mouse, the keyboard, the gamepad or any
other interaction device at their disposal. Addition-
ally, even wireless devices, they are normally placed
at physical location, breaking the virtual experience
of the user. New interaction devices have been devel-
oped to solve this issue, such as, walking platforms,
specific hand input devices, haptic gloves, etc. These
solutions may be a good choice for gaming purposes,
in which a higher level of activity is necessary or rec-
ommendable. However, depending on the application
or final user these devices may not be suitable. For
example, for educational purposes, with school stu-
dents, it seems more likely for them to use their own
computer screen and keyboard.
This work studies some interaction possibilities
of HMDs in a natural seated position. To analyse
the challenges of interacting with HMDs we con-
sider three universal interaction tasks within 3D or
VE applications (Hand, 1997), (Bowman et al., 2001),
(Bowman et al., 2004) which Jankowski, J. and Ha-
chet, M. (Jankowski and Hachet, 2013) defined as:
Navigation: Related to the motor task of moving
the viewpoint through an environment. If it in-
cludes a cognitive component, it is referred to as
Selection and Manipulation: Related to the tech-
niques of choosing and picking an object and
specifying its position, orientation, and scale.
System Control: Related to the communication
between user and system which is not part of the
For the use case presented in this work, we will study
the methodology followed when planning the exper-
iments and the decisions taken regarding these three
tasks according to our work experience.
This paper is organized as follows. Section 2 anal-
yses the related work regarding interaction in VEs and
hand gesture recognition. Section 3 explains the in-
teraction devices selected for the Setup of this work.
Section 4 describes the fire warden use case devel-
oped and analyses each of the interaction tasks. The
final section is about conclusions and future work.
Diez H., Moreno A. and Garcia-Alonso A.
Interaction Challenges in the Development of a Fire Warden VR Training System using a HMD.
DOI: 10.5220/0006133400840091
In Proceedings of the 12th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2017), pages 84-91
ISBN: 978-989-758-229-5
2017 by SCITEPRESS Science and Technology Publications, Lda. All rights reserved
Jankowski, J. and Hachet, M. (Jankowski and Hachet,
2013) presented a state of the art of non-immersive
interaction techniques for Navigation, Selection and
Manipulation, and System Control. This work also
summarizes the main 3D Web design guidelines.
Moya, S. et al. (Moya et al., 2014) analysed the
influence of different factors in locomotion control in
3D VEs. They designed an automatic locomotion sys-
tem and concluded that, for non-casual gamers, auto-
matic locomotion is preferred, whereas gamers prefer
to control locomotion themselves.
Regarding hand gesture recognition for HCI,
(Sharma and Verma, 2015) proposed the tracking of
six static hand gestures. They used images extracted
from recorded video streams at different distances and
positions. They are able to use their system to control
applications such as power point, media player, win-
dows picture manager, etc.
Manresa, C. et al. (Manresa et al., 2005) present a
real-time algorithm to track and recognise hand ges-
tures for interacting with the videogame. This algo-
rithm is based on three main steps: hand segmenta-
tion, hand tracking and gesture recognition from hand
features. The results of this work show that users can
substitute traditional interaction metaphors with their
low-cost interface.
Segura, I. et al. (Segura et al., 2007) developed a
simulator of construction machinery for safety train-
ing. In this work the visualization was implemented
as a chroma-key-based mixed reality system, combin-
ing a 3D VE, a real cabin interior, and some superim-
posed messages to the user. The overall impressions
from the testers were positive.
Goenetxea, J. et al. (Goenetxea et al., 2010)
presented an interactive and stereoscopic hybrid 3D
viewer. In this work they used Nintendo Wii control
and developed a dynamic gesture recognition proce-
dure used for interacting with the weather animations.
Regarding fire simulation, it has also been a re-
search topic in VR applications (Moreno et al., 2011),
(Moreno et al., 2012). The simulation of real-time in-
teractive fire is a challenging topic that has not been
solved yet.
In this work we propose to use a HMD combined
with usual interaction devices such as keyboards and
gamepads. Leap Motion Controller is used for hand
recognition. The following sections explain the setup
we have developed for a fire warden virtual training
use case.
In our use case, the tests are carried out autonomously
by a fire warden trainee. Therefore, it is not a collabo-
rative learning experience, as each user must be aware
of their own actions and decisions.
The setup used in these experiments is composed
of the following items:
Chair. One goal of our work is to keep the setup as
simple as possible. This is why we tried to limit
the amount of extra devices needed. The users
will interact in a seated position, needing no other
Computer. The recommended system require-
ments are: Windows 7 or newer, Compatible
HDMI video output, Intel i5, NVIDIA GTX 770
or greater.
Monitor. The final goal is for the user to visual-
ize the 3D VE with the HMD, but it is necessary
to have a monitor to make the first steps until the
virtual experience is launched and to get some ad-
ditional information after the training session.
HMD. We have used Oculus Rift DK2. The Ocu-
lus CV version or any other HMD brand could be
used, but the specifications of the computer might
require modifications to match the HMD require-
Keyboard. Anyone used to dealing with comput-
ers is familiarized to keyboards and there is no ex-
tra effort needed. Wireless or not is irrelevant, but
it should be placed in a known and fixed place be-
fore the VR training session. Bare in mind that
the user will not be able to see the real world with
the HMD on.
Gamepad. It is also a familiar device in video
games and it is intuitive for navigation purposes.
We have used the Logitech Rumblepad2.
Leap Motion Controller. This device provides
means for tracking the hand gestures performed
by the user. It was attached to the Oculus Rift
with the universal bundle. The latest Orion beta
software has been used (Orion, 2016).
3.1 Interaction Devices
This section discusses the interaction devices selected
for the use case presented in this work.
3.1.1 Keyboard
The keyboard is used to navigate through the VE us-
ing the up, down, left and right arrow keys. In tra-
ditional computers, they are a group of isolated keys,
Interaction Challenges in the Development of a Fire Warden VR Training System using a HMD
easily distinguishable from others. In more compact
devices such as laptops, the arrow keys despite being
closer to other keys are still easy to locate. The ad-
vantage with laptops is that the keyboard is attached
to the screen so it cannot change position. Sometimes,
WASD keys are also used for navigating. Normally,
this keys are used with the left hand leaving the right
hand free to interact with the mouse. This setup is
typical used in gaming (see Figure 1).
Both arrow keys and WASD keys can be used with
only three fingers of one hand, leaving the other hand
free. In this use case, the free hand is used to perform
the gestures defined to select and manipulate objects
in the scene. This gestures are captured by the Leap
Motion Controller.
3.1.2 Gamepad
Instead of using the keyboard to navigate, we have
experimented with a gamepad. In this case both hands
are needed to hold the gamepad and the user has to
let go of one hand to perform the hand gestures for
selecting and manipulating objects of the VE.
In most applications, the right stick is used for
navigating (forward, backward, left and right) and the
left stick for turning the camera view. This left stick
is not necessary as the user can look around by mov-
ing the head with the 360 tracking system offered by
the HMD. So, in the gamepad the user only has to use
the right stick, none of the other buttons are needed.
This makes it simple for the user, despite wearing the
3.1.3 Leap Motion
HMDs offer a really immersive experience. Users
feel they are actually in the VE they are watching
through the HMD device. This experience is so re-
alistic and convincing that users try to interact with
objects from the virtual world with their own hands.
However, regular interaction devices as the ones de-
scribed above do not allow this sort of interaction.
This is why we have chosen Leap Motion Controller
for our work. This device allows users to integrate
their hands into the virtual world and unlike other de-
vices, they require no practice or previous knowledge
from the user.
The Leap Motion Controller is mounted on the
front of the Oculus Rift glasses, as seen in Figure 2,
and whenever the users lift their hands to grab an ob-
ject from the virtual world, the virtual hands appear in
the virtual world, performing exactly the same move-
ments as the real hands.
This setup is more compact than other alternatives
using external trackers for gesture recognition.
This use case has been designed to train experts on
occupational hazard prevention and more precisely in
fire safety in buildings. The fire warden trainee has to
perform the tasks described below:
Locate the possible fires in the building.
Extinguish these fires.
Alert other people in the building.
Evacuate the building.
These tasks imply different levels of interaction
with the VE. In the next subsections we will explain
the methodology followed to decide which kind of in-
teraction devices to use, and we will explain the trial
and error process carried out for each of the classical
VR interaction categories.
4.1 Navigation
One of the tasks that the user has to perform is the
location of the possible fires throughout the building,
to discover these fires the user has to navigate through
a two-floored warehouse. The floors are connected by
two staircases which the user can use as many times
as needed.
Another task that implies navigation is the evac-
uation of the building. For these tasks the user has
to know the exits available in the building. Moreover,
the users are asked to put the fire out, to do that the lo-
cation of the extinguishers has to be found. The user
has to select them and head towards the fire.
These tasks imply wayfinding, as the user has to
have a cognitive overview of the building and has to
be able to make decisions. To ease this job, the appli-
cation displays some messages (images or text) on the
screen of the HMD. These messages are explained in
4.2 Selection
One of the goals of our fire safety training is to allow
wardens to learn how to use fire extinguishers in a
safe and efficient way. In real life a fire extinguisher
must be manipulated in a precise way and fires must
be put out following the PASS method (Safety and
Administration, 2016):
Pull the pin in the handle.
Aim the nozzle at the base of the fire.
Squeeze the lever slowly.
Sweep from side to side.
HUCAPP 2017 - International Conference on Human Computer Interaction Theory and Applications
Figure 1: Arrow Keys and WASD Keys in a Traditional Keyboard (left) and a Laptop (right).
Figure 2: Leap Motion Controller attached to Oculus Rift
Figure 3: Setup #1: Interaction with Keyboard and Leap
Motion Controller.
Figure 4: Setup #2: Interaction with Gamepad and Leap
Motion Controller.
We have made several tests to see which hand gestures
are most appropriate to simulate these actions in vir-
tual life and we have used the Leap Motion Controller
(LM) to track them.
In the first place, the warden has to grab a fire ex-
tinguisher. To allow this action in the fire safety train-
ing application, first the user has to navigate to one of
the extinguishers, and get close to it to a certain dis-
tance. We have set 60 cm as the maximum distance
to allow the user to grab the extinguisher. We have
chosen that distance as it is the standard length of an
adult’s arm (McDowell et al., 2008),(NASA, 2016).
Once the users are placed at reaching distance from
the extinguisher they have to extend their arm towards
it and grab it.
To represent this gesture, the first idea we thought
of was a “palm closing” gesture. This gesture is easily
trackable by the LM and it is easy to perform and un-
derstand by anyone: “I want to grab the extinguisher”.
However, we dismissed this gesture for two main rea-
sons. First, the PASS method indicates the user has to
pin in the handle. This action implies a more precise
movement of the fingers and we wanted to force the
user to do so. Second, as explained later, we selected
the “palm closing” gesture for opening and closing
doors or windows.
We want to enhance the interaction degree and the
learning experience. For that we designed indepen-
dent gestures for different actions. This way, the user
can easily internalize the meaning and consequences
of each virtual gesture in the real world. This is why
we decided to define the “pinch” gesture for grabbing
objects in the scene, in this case the fire extinguisher
(see Figure 5).
So, summarizing the full action process, the user
has to get close to the extinguisher, reach an arm to-
wards it and pinch it by holding the index and thumb
fingers together. The platform must detect a collision
between the extinguisher and the virtual representa-
tion of the user’s hand. If this happens, the extin-
guisher is moved towards the user and it is placed in
the right side of the visual field, representing that the
extinguisher has been grabbed and it is ready to be
used. The actions that the user can do with the extin-
guisher are discussed in the next subsection 4.3.
Interaction Challenges in the Development of a Fire Warden VR Training System using a HMD
Figure 5: “Pinch” gesture with Setup #1 (left) and Setup #2 (right).
Another goal, is to teach the warden how to evac-
uate the building safely. In this work, to do this, the
warden has to navigate to the closest exit and open the
door. We have chosen the “palm closing” gesture for
this action as it is the one we perform in real life to
grab door handles. The user has to approach a door
in the virtual scene and reach out the arm towards it.
As with the extinguisher selection, the user has to be
placed at least 60 cm. from the door.
At first, we thought on detecting the collision be-
tween the virtual hand and the door handle. We also
thought it would be recommendable to track the hand
turning down a certain angle as if turning a door han-
dle to open it. But these actions resulted confusing
and tedious. In the virtual scene, to detect a collision
between the hand and the handle, the warden had to
get very close to the door, which implied correcting
their position and performing the “palm closing” ges-
ture several times. Moreover, depending on the an-
gle degree the hand was turned, this gesture could be
confused with the “thumbs up” gesture. Finally, we
decided to simplify this action by allowing the door
to open by performing the “palm closing” gesture at
the right distance from an exit (see Figure 6).
Additionally, we have the same action to represent
a door by pulling it or by pushing it. Even if pulling
or pushing would affect in the cognitive process of
opening a door, we decided to omit this feature for the
sake of simplicity. Furthermore, it is expected that the
doors should be correctly designed, promoting “push-
ing” mode over “pulling”, as it may be required for
safety regulations.
4.3 Manipulation
In this use case, the fire warden must learn to manip-
ulate fire extinguishers. First, the user has to navigate
with the fire extinguisher towards the fire and manip-
ulate it following the PASS method. The correct dis-
tance to put out a fire safely is between three to one
meters. So if the user is further than 3 meters or closer
than 1 meter from the fire an alert text is displayed as
a HUD. Once the user is at the right distance, the user
has to aim at the fire, squeeze the nozzle and sweep
from side to side.
In real life, we would need both hands to manip-
ulate the extinguisher. However, in this virtual sce-
nario the user also has to interact with the keyboard
or gamepad. When using the keyboard, it is not con-
venient for the user to lift the fingers from the arrow
keys and with the gamepad, the user needs one hand
to keep holding it. To simulate the squeezing of the
nozzle, we decided to track one hand of the user per-
forming a “thumbs up” gesture (Figure 7), closing the
hand with the thumb heading up. We chose this ges-
ture because it is similar to the one users should per-
form in real life and to differentiate it from the “clos-
ing palm” gesture, in which the thumb is not visible.
Then, if the user is in possession of an extin-
guisher and performs the “thumbs up” gesture at the
right distance from a fire, a graphical particle system
representing the extinguishing jet will blast out from
the user’s hand position. If the user stops performing
the gesture, the particle jet stops. To increase the real-
ism, the noise of a real extinguisher is played through
the audio systems (integrated in the HMD or exter-
While performing this gesture the user also has to
move the hand from left to right to perform a sweep-
ing gesture.
Our Unity-based 3D engine (Unity3D, 2016) de-
tects whether a collision between the particle jet and
the fire has taken place. This way we check if the
user is aiming at the fire and sweeping the hand cor-
rectly. If there is a collision the fire is put out. We
define each fire as a collection of particle systems, so
they are put out gradually. If the user does not sweep
from side to side, the fire will not put out correctly.
Fire extinguishers contain around 10 seconds of ex-
tinguishing power. If the warden spends more than
that time trying to extinguish the fire it will probably
mean the sweeping is not being performed correctly.
In this case an alert HUD appears to remind the war-
den the PASS method.
HUCAPP 2017 - International Conference on Human Computer Interaction Theory and Applications
Figure 6: “Palm closing” gesture with Setup #1 (left) and Setup #2 (right).
Figure 7: “Thumbs up” gesture with Setup #1 (left) and Setup #2 (right).
Figure 8: Mode Selection.
4.4 System Control
This section explains the input messages we have de-
fined to communicate the user and the system. These
messages appear as HUDs (Heads-Up displays) on
the screen.
Where to place each HUD is also important, as
far as possible they should be integrated in the virtual
scene (Yao et al., 2014).
The user can begin the simulation in training mode
or in simulation mode (see Figure 8). The user selects
between these options with the LM.
During the simulation the user can visualize
a semi-transparent and non-intrusive 2D mini-map.
This map has been designed as an egocentric map.
The map shows the following signs to help the user:
“You are here” marker.
Figure 9: Upper Left: 2D mini-map. Upper Right: Timer.
Exit location signs.
Extinguisher location signs.
A timer is also visible at all times, so the user knows
how long it is taking to perform the exercise. These
HUDs are shown in Figure 9.
In the training mode, some green arrows are dis-
played on the 3D environment to show the user the
path to the closest extinguisher, once the user has se-
lected the extinguisher and put out the fire, again some
arrows are displayed to show the path to the clos-
est exit (see Figure 10). These signs are not shown
in the simulation mode, as the user should have ac-
quired knowledge about the environment in the train-
ing mode.
Interaction Challenges in the Development of a Fire Warden VR Training System using a HMD
Figure 10: Training Mode. Green arrows placed on the floor
guide the user to the exit.
In this work, a VR fire warden training system has
been presented. Its main objective is to provide VR
experiences to fire wardens in order to internalize
and comprehend the concepts behind the standard-
ized procedures that have to be followed when a fire
emergency arises in a building. In our preliminary
developments, a few actions have been tested as they
target the main interactions within VR environments:
Navigation for evacuation routes and fire finding; Se-
lection and Manipulation to interact with the extin-
guishers and doors; and System Control interaction
to display non intrusive 2D information as a HUD.
All the actions have been developed as a combination
of gestures recognized with the aid of Leap Motion
Controller and keyboard/gamepad to navigate the 3D
The seated position has been defined as a require-
ment from the very beginning as it is the most com-
fortable way of interacting with the VR environment.
The stand-up position poses additional problems like
i) problems with the cabling, ii) collisions with the
real-world furniture and iii) anxiety regarding the
cognitive disassociation of the virtual position and the
real position, specially when stairs are included in the
scenarios. The seated position is also recommendable
if the interaction is going to last a long period of time.
More trials and validation of the VR setup is
needed. However, we have found that the election of
the keyboard or the gamepad is a matter of personal
preferences. In any case, we have reports about the
necessity to adapt the keys of the keyboard or buttons
of the gamepad to each personal preference.
Some users found some limitations in the VR nav-
igation. They report that turning commands conflict
with looking to the sides in the VR environment. Nev-
ertheless, it is a matter of time to get the users ac-
quainted with the navigation system.
As future work, the following prototype will ex-
periment alternatives to solve the problems reported
by the users. A more extensive evaluation will be car-
ried out.
The extension of the prototype to other users and
purposes is also under consideration. The presented
use case could be adapted easily to children in order
to show them basic information about what they have
to do when a fire emergency arises.
After the preliminary implementation, we are
planning to introduce a different use case, oriented to
the visualization of 3D models on the Web. This use
case is very different from the one presented in this
work and therefore, it will provide complimentary in-
formation about the VR techniques in a different VR
Bowman, D., Kruijff, E., LaViola Jr, J. J., and Poupyrev,
I. (2004). 3D User Interfaces: Theory and Practice,
CourseSmart eTextbook. Addison-Wesley.
Bowman, D. A., Kruijff, E., LaViola Jr, J. J., and Poupyrev,
I. (2001). An introduction to 3-d user interface de-
sign. Presence: Teleoperators and virtual environ-
ments, 10(1):96–108.
Goenetxea, J., Moreno, A., Unzueta, L., Galds, A., and Se-
gura, l. (2010). Interactive and stereoscopic hybrid
3d viewer of radar data with gesture recognition. In
Graa Romay, M., Corchado, E., and Garcia Sebas-
tian, M., editors, Hybrid Artificial Intelligence Sys-
tems, volume 6076 of Lecture Notes in Computer Sci-
ence, pages 213–220. Springer Berlin / Heidelberg.
Hand, C. (1997). A survey of 3d interaction techniques.
In Computer graphics forum, volume 16, pages 269–
281. Wiley Online Library.
Jankowski, J. and Hachet, M. (2013). A survey of inter-
action techniques for interactive 3d environments. In
Eurographics 2013-STAR.
Manresa, C., Varona, J., Mas, R., and Perales, F. (2005).
Hand tracking and gesture recognition for human-
computer interaction. ELCVIA Electronic Letters on
Computer Vision and Image Analysis, 5(3):96–104.
McDowell, M. A., Fryar, C. D., Ogden, C. L., and Flegal,
K. M. (2008). Anthropometric reference data for chil-
dren and adults: United states, 2003–2006. National
health statistics reports, 10(1-45):5.
Moreno, A., Segura,
A., Korchi, A., Posada, J., and Otaegui,
O. (2011). Interactive urban and forest fire simulation
with extinguishment support. In Advances in 3D Geo-
Information Sciences, pages 131–148. Springer Berlin
Moreno, A., Segura,
A., Zlatanova, S., Posada, J., and
ıa-Alonso, A. (2012). Benefit of the integration
of semantic 3d models in a fire-fighting vr simulator.
Applied Geomatics, 4(3):143–153.
Moya, S., Grau, S., and Tost, D. (2014). First-person loco-
motion in 3d virtual environments: a usability analy-
sis. J. UCS, 20(7):1026–1045.
HUCAPP 2017 - International Conference on Human Computer Interaction Theory and Applications
NASA (2016). Antrhropometry and biomechanics.
Orion (2016). Orion beta sw.
Safety, T. O. and Administration, H. (2016). Fire protection
and prevention.
Segura, l., Moreno, A., Brunetti, G., and Henn, T. (2007).
Interaction and ergonomics issues in the development
of a mixed reality construction machinery simulator
for safety training. In Dainoff, M., editor, Ergonomics
and Health Aspects of Work with Computers, volume
4566 of Lecture Notes in Computer Science, pages
290–299. Springer Berlin / Heidelberg.
Sharma, R. P. and Verma, G. K. (2015). Human computer
interaction using hand gesture. Procedia Computer
Science, 54:721 – 727.
Unity3D (2016). Unity3d.
Yao, R., Heath, T., Davies, A., Forsyth, T., Mitchell, N., and
Hoberman, P. (2014). Oculus vr best practices guide.
Oculus VR.
Interaction Challenges in the Development of a Fire Warden VR Training System using a HMD