MOBILE INDOOR AUGMENTED REALITY
Exploring Applications in Hospitality Environments
Ana M. Bernardos, Jesús Cano, Josué Iglesias and José R. Casar
ETSI Telecomunicación, Universidad Politécnica de Madrid, Madrid, Spain
Keywords: Augmented reality, Ubiquitous computing, Mobile applications, Tourism applications.
Abstract: Augmented reality (AR) is been increasingly used in mobile devices. Most of the available applications are
set to work outdoors, mainly due to the availability of a reliable positioning system. Nevertheless, indoor
(smart) spaces offer a lot of opportunities of creating new service concepts. In particular, in this paper we
explore the applicability of mobile AR to hospitality environments (hotels and similar establishments).
From the state-of-the-art of technologies and applications, a portfolio of services has been identified and a
prototype using off-the-shelf technologies has been designed. Our objective is to identify the next
technological challenges to overcome in order to have suitable underlying infrastructures and innovative
services which enhance the traveller’s experience.
1 INTRODUCTION
The advances of personal mobile technologies have
made possible to create new concepts to interact
with the environment and the objects in it through
mobile devices. In this direction, augmented reality
(AR) relies on combining and superimposing virtual
information over the real world, providing the user
with extra (even real time) computer-based
information about resources, objects or points of
interest.
The ‘augmented reality’ concept is not new, but
has been revisited since the sixties, when Sutherland
(Sutherland, 1968) designed a head-mounted display
tracked by both mechanical and ultrasonic trackers.
Nevertheless, the term, as it is, is in use since 1992
(Caudell and Mizell, 1992). Nowadays, the
generalization of high-resolution cameras and
embedded compasses and inertial systems in mobile
devices has created the technological breeding
ground for the democratization of mobile AR
services.
In this paper, we explore the possibilities of
mobile augmented reality as interface with indoor
(smart) spaces. In particular, we focus on the
potential implementation and uses of indoor AR in
tourism settings such as hotels. The paper is
organized as follows. Section 2 reviews the state of
the art of AR in mobile devices: technologies,
applications and developing tools. Section 3 gathers
some AR service concepts we are currently handling
to create the Hotel of the Future in the THOFU
project. Section 4 describes our design for a testbed
in our lab to experiment with indoor AR. Finally,
Section 5 concludes the paper with some comments
on open challenges.
2 MOBILE AUGMENTED
REALITY: STATE-OF-THE-ART
In order to deploy AR services, it is necessary to
define, in real time and continuously, the spatial
relationship between the user - who is carrying or
wearing the visualization display - and the objects or
reference points in the ‘user’s coverage area’.
Distance and orientation between the target and the
user will be principal inputs for AR service delivery.
In order to calculate this spatial relationship and
track it, three types of techniques are currently used:
a) those based on image recognition – vision-based
tracking; b) those working on inertial and
positioning data – sensor-based tracking and c)
hybrid tracking, combining both methods. Readers
interested in tracking technology can resort to (Zhou
et al., 2008) for a wide review of existent
technology.
While talking about mobile AR, the first group
of techniques relies on the use of a camera to
232
Bernardos A., Cano J., Iglesias J. and Casar J..
MOBILE INDOOR AUGMENTED REALITY - Exploring Applications in Hospitality Environments.
DOI: 10.5220/0003400102320236
In Proceedings of the 1st International Conference on Pervasive and Embedded Computing and Communication Systems (PECCS-2011), pages
232-236
ISBN: 978-989-8425-48-5
Copyright
c
 2011 SCITEPRESS (Science and Technology Publications, Lda.)

continuously ‘snapshot’ the target object and process
the image to estimate the position, orientation and
movement of the visualization display with respect
to the target object. Direct object recognition may be
difficult and unstable, but the deployment of
markers improves the performance of the
recognition system. Markers may be easily
recognizable by the user (e.g. bidimensional codes -
stuck or printed on the object) or hidden inside an
image. In general, lighting and focus related
problems limit the performance of AR services using
this approach.
The second group of techniques works with
algorithms that fuse orientation, movement and
location estimates from inertial and positioning
systems to physically reference the user to the target
object. When outdoors, GPS offers sufficient
accuracy to locate the user, while compasses may
provides orientation and accelerometers (and gyros,
if existent) allow estimating the device’s relative
inclination. When indoors, radio-based positioning
systems (propagation channel model or fingerprint-
based ones) are usually offering an average accuracy
that is not enough to deliver AR functionalities.
Thus, it is necessary to switch to costly and dense
infrastructures (e.g. based on ultrawideband,
ultrasounds or infrared localization systems),
typically complemented with user-worn devices, in
order to achieve centimetre-level accuracy.
Table 1 gathers a sample of currently available
mobile AR services, classified depending on the
operative system they run on top of and the
technology they use to estimate the relative position.
As the reader may notice, some of the applications
fuse the use of camera and location technologies
(hybrid techniques). Most of the proposals are
designed for outdoors, and when indoors they are
relying in vision marker-based tracking.
With respect to the state-of-the-art of developing
tools, nowadays it is possible to find proposals using
a) image recognition and b) GPS information for
tracking. Following there is a list of some of the
available toolkits for different mobile platforms.
ARToolKit is a tool that facilitates target
tracking and virtual object interaction through
images; versions are available both for iPhone,
Android and Windows Mobile (e.g. ARToolWorks,
Studierstube Tracker), and a limited version for Java
and C# is also available (NyARToolkit). Qualcomm
offers a SDK to develop Android AR applications.
Still in beta, it is designed to develop image based
AR applications, facilitating basic features such as
identification and target tracking, and interface
features, such as interaction with virtual buttons.
D’Fusion Mobile provides a platform for iPhone and
Android, which allows marker-based object
recognition, together with face recognition and
accelerometers and localization support. Unifieye
Mobile facilitates the development of AR marker-
based applications for iPhone, Android, Symbian
and Windows Mobile, additionally working on GPS
and compass.
Table 1: Review of selected mobile AR applications.
Note: POI stands for Point of Interest.
O.S. Sensors Description
Layar
iPhone,
Android
GPS,
compass
& acc.
AR browsers.
Info about
POIs.
Wikitude
iPhone,
Android,
Symbian
GPS,
compass
& acc.
Google
Goggles
iPhone,
Android
Camera,
GPS
Image-based
web search
(monuments,
books, brands,
etc.)
Space
InvadAR
iPhone,
Android
Camera AR Game
Google Sky
Map
Android
GPS,
compass
Sky
constellations
Star Chart iPhone
GPS,
compass
Madrid
nearest Metro
iPhone
GPS,
compass
Nearer metro
stations
Cyclopedia iPhone
GPS,
compass
Wikipedia
info about
POIs
SnapShot
Showroom
iPhone Camera
Preview of
furniture in
rooms
Augmented
Car Finder
iPhone
GPS,
compass
Parking
helper
buUuk
iPhone,
Android,
Symbian
GPS,
compass
Restaurant
guide
Le Bar Guide iPhone
GPS,
compass
The Virtual
Public Art
Project
iPhone,
Android
GPS,
compass
Virtual
sculptures on
real settings
(over Layar)
Word Lens iPhone Camera
Signs
translator
Mentira iPhone
GPS,
compass
AR game to
learn Spanish
The Layar browser, available both for iPhone and
Android, can be used to create new AR applications
based in GPS. This tool is only suitable to customize
the information about points of interest available in
MOBILE INDOOR AUGMENTED REALITY - Exploring Applications in Hospitality Environments
233

Layar. Mixare, Wikitude and junaio are similar
alternatives to Layar for Android and/or iPhone.
Wikitude provides an API that allows integrating its
localization system in an external application. ARIS
(Augmented Reality and Interactive Storytelling) is
an open source tool to create outdoors AR
educational games. For Windows Phone 7, the
available SDK for Visual Studio and its
documentation makes possible to access embedded
sensors (accelerometers and GPS), although it is not
possible to access the camera. This restriction does
not exist in previous versions of Windows Mobile.
Mono allows using C# code in environments which
are not originally prepared for that, e.g. iPhone,
without losing control over the device’s APIs. It is
an alternative when using C# is an option either for
efficiency, knowledge or reusability.
Most of the available tools are designed for
iPhone or Android. For any of these operative
systems, the choice depends on the type of target
application. Our objective is to apply mobile AR
concepts to be used indoors, in a very defined
domain. Next Section details the type of applications
we are considering.
3 INDOOR MOBILE AR
APPLICATIONS
FOR HOSPITALITY SETTINGS
The THOFU project (Technologies for the HOtel of
the FUture) is a cooperative Spanish research
project; 35 entities research technologies that may
serve to configure a new offering of context-aware
user-centric services in advanced hospitality
infrastructures. Within this application framework,
AR is considered to be a relevant concept enabling
to build a new service offering.
To date, commercial mobile AR applications for
tourists are ready to be used outdoors. For example,
the Museum of London is providing StreetMuseum,
an iPhone application superimposing information
about old London all over the city. When facing
indoors, to the best of our knowledge experiences
are still prototypes (limited in time and space): for
example, the iTacitus project has delivered AR
applications for the Palace of Venaria in Turin
(Zoellner et al., 2009) – to see how frescos on the
walls once appeared – or to show how the court
inside the Winchester castle was. The Louvre-DNP
Museum Lab Project has resulted in an Ultra-
Mobile-PC AR museum guide using markerless
tracking (based on Ubisense ultrawideband) (Miya-
shita et al., 2008).
THOFU aims at exploring the possibilities of AR
in hotels to enhance the visitor’s experience, in order
to:
 Become familiar with the room: mobile AR may
be used to provide additional information about
standard objects in hotel’s rooms. The guest may use
an AR application to discover resources in the room,
for example where the strongbox is located. The fix-
line phone may be augmented with its agenda and
additional information about pricing; the television
may be augmented with the programmes and the pay
TV offer. Pillows may show the pillow menu, soaps
and gels may show their composition and furniture
or decorations may offer information about their
design or even information about how to acquire
them. Additionally, AR may enhance the way we
interact with smart home controls, e.g. allowing to
visualize the room temperature in graphical mode
and modify it when pointing the mobile device to the
air conditioning.
 Improve Access to Safety & Emergencies
Information: the traveller may receive information
about the electricity system when pointing at a plug
(voltage, connector type, where to acquire a current
adaptor, etc.). Information about emergency way-
outs and procedures may be easily consumed on an
AR application too.
 Facilitate Navigation in Complex Environments:
Virtual sings may be superimposed to real views to
guide the user towards his destination. It is important
to note that the navigation system should perform
well both indoors and outdoors, and even in special
areas such as parkings. Combined with geotagging,
AR navigation may offer services such as car search.
The combination with a well-situated marker
catalogue – which may facilitate searching or typing
the destination - may enhance the application use.
 Offer a Different Service Experience: mobile AR
makes possible to offer additional information about
the available dishes in a menu, their composition or
their nutritional features. For example, it is possible
to visualize the menu when pointing at the table or
the drink offer by pointing at the bar.
 Provide Configurable Virtual Decoration. For
example, virtual exhibitions may be offered in some
spaces of the hotel. These exhibits will need to be
‘visited’ through the mobile device. Additionally,
the guest may have the option of virtually
refurbishing some customizable items in his room,
in order to attach virtual data to real objects (e.g. the
user may want to check the weather when pointing
some objects in the room).
PECCS 2011 - International Conference on Pervasive and Embedded Computing and Communication Systems
234

 Facilitate Interaction between Staff and Clients
and Improve Processes: face recognition-based AR
applications may speed checking up by helping the
staff to recognize frequent customers’ faces in order
to personalize treatment.
 Motivate the User to Discover the Hotel
Surroundings: from balconies and terraces mobile
AR applications may serve to visualize nearby
resources and tourism information.
Additionally, it is possible to deliver mobile AR
applications for maintenance tasks. For example, the
sensing infrastructure may be controlled through an
application visualizing the state and the levels from
sensors just superimposed on the real world where
they are deployed.
With these applications in mind, we want to
configure a testbed to try new mobile AR concepts
(for hospitality) on top of off-the-shelf underlying
technologies. Next section describes our
technological choice and first trials.
4 PROTOPYTING
AN INDOOR AR TESTBED
Our final objective is to build up an indoor AR
sensor-based testbed. The system should recognize
tagged objects and points of interest (PoIs) to enable
the list of applications in the previous section. We
do not discard to combine the solution with vision-
based complementary algorithms to enhance the
system performance, but we firstly focus on setting
up a solution for the sensor-based approach. If
accurate, we consider that it is much easily scalable
(e.g. including new PoIs does not require any extra
modification of the algorithm base) than the vision-
based one. Additionally, it can set up the basis for
new interaction schemas (e.g. pointing computing)
helping to spread AR use.
From the review of developing tools, we have
chosen to use the open source Mixare. Mixare works
over Android. We have opted for Android due to its
standard, easy and well documented APIs to access
sensors, apart from the availability of advanced
hardware and the growing market share.
We have modified its native GPS localization
API to make it work with our indoor positioning
systems. PoIs have been included in JSON format
(JavaScript Object Notation, a light format for data
exchange) in a file, adapting them to use the
available standard method to load information in
Mixare.
Mixare works on WGS84 datum to represent co-
ordinates (the system used by GPS receivers). The
coordinates of the PoIs have been integrated in GPS
format in order to handle them with default methods
(nevertheless, some conversions from UTM to GPS
have been needed).
Finally, interface methods have been modified to
adapt the position of the plotted markers referencing
PoIs in order to enhance visualization. Figure 1
contains a screenshot of the application deployed
over a HTC Hero Android device, showing the name
of the POIs and the estimated distance to the
visualization device.
Figure 1: Screenshot of our indoor AR object tagging
application based on Mixare.
As previously stated, indoor AR applications are
very demanding in terms of accuracy - errors are
easily perceived by the user (distances are shorter
and easy to estimate); thus, service experience may
be ruined if underlying localization technologies
underperform. Our RF positioning systems - based
on WiFi, ZigBee and Bluetooth, see e.g. (Aparicio et
al., 2008) - provide enough accuracy to perform
reliable estimation of the zone/area where the user is
staying (error varies between 2 and 3 meters). As
this solution is not suitable for indoor AR and we are
looking for a non-pricey infrastructure with
minimum hardware needs, we initially have opted to
deploy a set of reference positions from which the
user is capable of getting the information of all the
tagged PoIs around him. These positions identified
by pressure mats connected through a ZigBee
communication channel to a sensor network
covering the deployment area. Of course, this is a
suboptimal solution because it limits the service
availability and restricts natural interaction, but it
allows handling concept testing.
5 CONCLUSIONS
Most of commercial AR applications have been
designed to work outdoors, while indoor settings
MOBILE INDOOR AUGMENTED REALITY - Exploring Applications in Hospitality Environments
235

present new challenges not only for accurate
positioning, but also for the AR application concept
itself.
The lack of stable and accurate indoor location
estimation usually requires ad-hoc localization
deployments, typically expensive and still unfeasible
in normal settings. For this reason, it is possible to
think that the most adequate approach to adopt for
indoor AR is vision-based. Nevertheless, these
systems are unstable enough to require the user to
make several attempts (focusing the mobile camera
once and again) before getting a satisfactory result
(due to light, camera diversity and user skills). Apart
from that, they usually require to deploy markers to
guarantee good detection. Markerless tracking is still
an open challenge.
Considering scalability, vision-based techniques
scale worse than sensor-based ones, as recognizing a
new object requires to have it characterized in the
image database. Additionally, image search is
usually a resource-consuming task, which may have
to be performed (or supported) by an external
infrastructure.
Then, even if a reasonable amount of context-
aware services may perform well using symbolic
location, there is still a clear need for accurate
inexpensive indoor positioning systems to
implement sound AR services. As embedded inertial
systems are increasing their quality and diversity
(gyroscopes have been recently included in some
mobile devices), the solution may rely on fusing data
coming from many sources of information, which
may be opportunistically deployed depending on the
environment.
Using mobile AR should represent a step
forward towards natural interaction, so the services
to be built on it should make a difference to other
interface options. Exploring new interaction
concepts to capture the real objects or POIs - such as
using the mobile device as a remote control (not
needing to use the camera) - may facilitate user
adoption.
ACKNOWLEDGEMENTS
This work is being supported by the Government of
Madrid under grant S2009/TIC-1485 and the
Spanish Ministry for Science and Innovation under
grant TIN2008-06742-C02-01. Additionally, the
application area is motivated by the THOFU CENIT
Project, funded by the Centre for the Development
of Industrial Technology.
REFERENCES
Aparicio, S., Pérez, J., Bernardos, A. M., Casar, J. R.,
2008, A fusion method based on Bluetooth and
WLAN Technologies for Indoor Location, IEEE Int.
Conf. in Multisensor Fusion and Integration for
Intelligent Systems, Seoul.
Caudell, T. P., Mizell, D. W., 1992, Augmented Reality:
An Application of Heads-Up Display Technology to
Manual Manufacturing Processes, Proceedings of
1992 IEEE Hawaii International Conference on
Systems Sciences, pp 659-669.
Feng, Z., Duh, H. B.-L., Billinghurst, M., 2008, Trends in
augmented reality tracking, interaction and display: A
review of ten years of ISMAR, 7th IEEE/ACM
International Symposium on Mixed and Augmented
Reality, pp. 193 – 202.
Mixare, http://www.mixare.org
Miyashita, T., Meier, P., Tachikawa, T., Orlic, S., Eble, T.,
Scholz, V., Gapel, A., Gerl, O., Arnaudov. S.,
Lieberknecht, S., 2008, An Augmented Reality
Museum Guide, Proceedings of the 7th IEEE/ACM
International Symposium on Mixed and Augmented
Reality, pp. 103-106.
Sutherland, I., 1968, A Head-Mounted Three Dimensional
Display, Proceedings of Fall Joint Computer
Conference, pp. 757-764.
THOFU, http://www.thofu.es
Zoellner, M., Keil, J., Drevensek, T., Wuest, H., 2009,
Cultural Heritage Layers: Integrating Historic Media
in Augmented Reality, 15
th
International Conference
on Virtual Systems and Multimedia, pp. 193-196.
PECCS 2011 - International Conference on Pervasive and Embedded Computing and Communication Systems
236