Design and Development of the First Prototype of a Social, Intelligent

and Connected Help Desk

Simona D’Attanasio

, Thierry Sotiropoulos

and Rachid Alami

2 a

GEMIA Departement, Icam site of Toulouse, Toulouse, France

LAAS-CNRS, Université de Toulouse, CNRS, Toulouse, France

https://www.researchgate.net/profile/Simona_Dattanasio

Keywords: Human-machine Interface, Instrumented Help Desk, OpenPose, Modular Architecture, Tourism Application.

Abstract: This paper presents an innovative architecture of help desk in the tourism environment. The aim of the work,

performed with a professional woodworker manufacturer, is to design an interactive counter, integrating the

methods and the technologies of robotics. The idea is to provide customers with a new experience, based on

an intelligent and social interaction within a connected environment, using wood as the main material. In

order to achieve this objective, modular hardware and software architectures have been designed and the first

prototype is under development. The paper focuses on the description of the architecture of this prototype and

on the first results obtained.

1 INTRODUCTION

A help desk, or counter, is not only a piece of

furniture, it is the entry/exit point of commercial,

administrative or cultural entities, public or private,

having a physical access. The help desk represents the

physical interface where customers and service

providers in tourism get in touch and interact.

Examples of service providers are hotels, restaurants,

tourist offices and so on.

In this context, modern technologies can provide

new forms of interactions and ways to deliver

services, thus contributing to enhance customer

experience, or user experience (UX). UX consists in

providing a usable, useful and enjoyable service or

product (Rogers, Sharp and Preece, 2011). UX is a

matter of emotions before, during and after use. It is

how a product feels (Allabarton, 2018).

A lot of work can be found in literature about

pieces of furniture or more complex systems, able to

interact with customers through technology.

Innovative kiosks designed as information points

have been the object of many publications. Niculescu,

Yeo and Banchs (2016) propose a device for

shopping mall, based not only on tactile but also on

vocal interfaces, while in the work of Bergweiler,

Deru and Port (2010) tourists can even share

https://orcid.org/0000-0002-9558-8163

information through a similar tactile device. Focused

on the development of a social conversational agent,

the work of Cassell et al. (2002) proposes the analysis

of speech, gaze, gesture and body posture to better

give to the visitor the demanded direction. Finally, the

system designed for smart cities and described by

Gomez-Carmona, Casado-Mansilla and Lopez-de-

Ipiña (2018) points out the importance of taking into

account emotion bond, personalization and

technology appropriation by the user in kiosk design.

Other than kiosks, the work described in Omojola

et al. (2000) presents a table installed in a museum

projecting contents from the ceiling. Tagged objects

enable information selection. The interesting idea of

this project is to provide the visitor with an intuitive

and unobtrusive information interface non neglecting

the aesthetic of the visual appearance. Another

interesting design approach to furniture design for

workspaces is proposed by Campos, Ehrenberg and

Campos (2018). In this work it is pointed out how a

multisensory approach and ergonomics can affect

productivity in a workplace.

An original work and application is proposed by

Schafer et al. (2018), that presents an instrumented

room to help fighting against illiteracy. This room is

used by librarians to engage a group of children in a

multi-sensory read-aloud of a printed picture book.

120

D’Attanasio, S., Sotiropoulos, T. and Alami, R.

Design and Development of the First Prototype of a Social, Intelligent and Connected Help Desk.

DOI: 10.5220/0008162601200127

In Proceedings of the 3rd International Conference on Computer-Human Interaction Research and Applications (CHIRA 2019), pages 120-127

ISBN: 978-989-758-376-6

The system can be programmed to play sounds, lights

and movements to create scenes. The idea of plunge

the user into an atmosphere is very inspiring, but the

user interaction with the system is limited to a

touchscreen tablet.

Some of these technologies are now industrial

products. Restaurants, bars and hotels integrating

interactive environments and even robots can be

found all over the world. The Tipsy robot in Las

Vegas, for example, is an installation integrating two

industrial Kuka robots, that prepare cocktails created

by customers (http://thetipsyrobot.com). A social and

connected environment allows customers to share

recipes, to rate favourite and some other features. In

the Inamo restaurant in London (https://www.inamo-

restaurant.com) the atmosphere can be tuned thanks

to a 360° projection of a real spot or of a completely

virtual world. Tactile bar counters, reactive to bottles

and glasses, are today available, as the products

proposed by Kineti (http://kineti-technologies.com).

In Asia, high-tech restaurants and hotels even employ

robots as waiters or receptionists. As an example, it is

possible to be served by robots at the Chinese

Freshippo supermarket. Most of these industrial

products show poor multimodal interaction and

favour one technology. They have a reduced field of

applications and are difficult to use if either the

environment or the customer’s need changes.

Moreover, they are extremely costly, therefore their

spread is very limited.

The system proposed in this article aims to

integrate existing low cost technologies in a modular

and scalable architecture, able to provide multiple

services. The final objective of such a system is to

enhance UX, by creating an emotional bond with the

customer. This is achieved by guiding the customer

through a unique experience by means of multiple

multimodal interactions, carefully integrating

aesthetic design and choice of materials. Recent

studies have indeed investigated that wood-based

interior products improve customer touch experience

(Bhatta et al., 2017) and have positive physiological

effects (Ikei, Song and Miyazaki, 2017). As a result,

our system design is performed in close collaboration

with a professional woodworker specialised in

counter manufacturing. In addition, the participation

to the project of a design professional contributes to

ergonomics and aesthetic of the final outcome.

None of the system found in literature or among

commercial products deals with all these aspects. The

application addressed by this system is the tourist

field, but the concept can be easily adaptable to any

environment integrating a help counter.

The paper is organized as follows. In Section 2 the

basic concept is presented, while in Section 3 the

main technological choices are introduced. Section 4

describes the prototype and Section 5 presents the

first results. Section 6 concludes.

Since the acronym of this project is CISC

(Comptoir Intelligent Social Connecté, which means

intelligent social connected counter), we will refer to

the help desk as the CISC counter or simply as CISC.

2 THE BASIC CONCEPT

CISC consists of several physical interactive

counters, called modules, organized within a zone,

called interaction area. The main idea is that the

customer’s interaction with CISC is supported not

only locally at each module, but all over the area. This

enables a smooth and intuitive interaction all along

the customer’s path, proving a higher quality UX.

Figure 1: CISC basic concept.

More precisely, our concept is illustrated in Figure

1. As the customer enters the interaction area, his/her

movements are tracked in real-time. This information

can be used to help the customer to find the way to

terminals, to optimize system layout and to improve

customer flow. As the customer enters the interaction

area, an agent is allocated to him/her. The agent is the

unique entity created by the system for that customer.

It guides and assists the customer during his/her

interaction with CISC.

Within the interaction area, nearby a non-

interactive help desk managed by a human staff

member, there is at least one module, called master.

Depending on the number of customers to be

Design and Development of the First Prototype of a Social, Intelligent and Connected Help Desk

121

welcomed, additional masters can be installed. One or

more physical secondary modules, called terminals,

are also installed. This modular architecture allows

the adjustment of the system configuration according

to the environment (amount of attendance, building

configuration, cost, etc.).

Customers can access the area from any point, but

the first action they should perform is to head to one

master, in the same way a customer looks for a free

staff member in a tourist office. A dynamic back-

counter behind masters aims to draw the customer’s

attention before the beginning of the interaction. It is

also used to enhance system communication with the

customer during the interaction. Masters present to

the customer the services available in the area

(including those provided by staff members) and

record customer preferences: for example, they

enable to choose the language. Preferences are stored

by the system and transferred to terminals, that

provide services. That’s why terminals do not activate

themselves in front of a customer that hasn’t

registered to a master. Terminals are stand-alone

modules that can be plugged or unplugged to the

system and that communicate with it. The technology

that they integrate depends on the provided service(s).

All modules, masters as well as terminals, are

“console” shaped (bended table), as it can be

observed in Figure 2.

Figure 2: Example of a console shaped master designed by

the professional designer of the project.

In this way, all modules are completely accessible

for wheelchairs. They are designed by the

professional in the field of interior design for

commercial and business environments, partner of the

project. Moreover, modules are entirely

manufactured by the professional woodworker, also

partner of the project. High quality wood and other

special materials are carefully selected and

assembled.

The presented concept has the main objective of

enhancing UX and of providing effective help to staff

members. That’s why design specifications focus on

the following key aspects:

- An intelligent desk, able to deliver a customized

service by identifying the customer (with no

connection to his/her real identity), by storing

interests and preferences. Staff members can also

use the data to improve offers and services.

- A social desk, able to pertinently react to

customer’s behaviours such as hesitation, doubt,

immobility. It also assists the customer during the

interaction.

- A connected desk, able to create a bond between

the customer and local environment (touristic

attractions, product, etc.). Wood is chosen as the

main material to convey emotions and create this

bond.

These key design objectives have driven our main

technological choices.

3 MAIN TECHNOLOGICAL

CHOICES

The CISC concept relies on a modular and scalable

design, based on a multi masters/multi terminals

hardware architecture. For the software architecture

we use ROS Robotic Operating System. The reason

is the need of:

- Modularity. The CISC design and configuration

are intrinsically modular. The number of modules

as well as the services available are always

different, according to staff and customer needs.

- Parallelism. Many people (customers and staff

members) can interact with CISC at the same time

and the system has to respond in real-time.

- Technology adaptability. CISC integrates existing

technologies and is meant to evolve over time.

Many input/output devices are already integrated

in ROS and are immediately available.

3.1 An Intelligent Desk

In order to be able to identify the customer and to

track his/her path within the interaction area, a badge

based on active RFID (Radio Frequency

IDentification) technology has to be collected by the

customer before entering the area. The action of

collecting a badge has a precise meaning for the

customer interacting with CISC. It is a choice, a will

CHIRA 2019 - 3rd International Conference on Computer-Human Interaction Research and Applications

122

of beginning an interaction that ends when the badge

is given back. The work of Fossdal and Berg (2016)

directly inspired us in building this scenario. The

badge is necessary to activate the different modules

and allow the recording of the customer activity. The

badge is the agent, previously introduced. The active

RFID-based system is developed by a company

specialized in IoT (Internet of Things), partner of the

project. An advantage of this method with respect to

image processing performed by cameras is the

robustness of customer tracking in crowded

environments. Another advantage is that the customer

can remain anonymous. The system doesn’t store any

personal data without consent in compliance with the

EU General Data Protection Regulation (GDPR). A

customer can record only willingly the profile for a

further visit.

3.2 A Social Desk

In order to be able to interact in a natural way with the

customer, a main strategy based on Artificial

Intelligence techniques is under development. At the

master, a low cost camera captures a video stream that

is analysed by the OpenPose library (Cao et al.,

2018). This open-source library is based on a neural

network algorithm and provides in real-time and in a

robust way the position of arm and body segments.

The idea is to detect hesitation, immobility or

inactivity, bad manipulation caused by accident

(error) or voluntary (not “playing the game”).

3.3 A Connected Desk

The main application of our desk is assistance in the

tourism field. Since the system is highly modular and

reconfigurable, it can be installed in tourist offices,

hotels, restaurants, museums and so on. The desk can

operate autonomously or coexist with human

operators, according to the specific needs of the

environment. The idea is to create a connection

between the customer and the touristic local

activities, services and/or products by means of

multiple human-machine interfaces, having multiple

interaction modalities, e.g. tactile, gestural, visual. In

order to provide an original UX, our system integrates

wood in these interfaces.

4 THE PROTOTYPE

The first prototype of CISC is under development.

Figure 3 shows the overall architecture of the system.

The use case addressed by the project is the tourist

office. A master with a moving wall and two

terminals are placed within the interaction area.

Figure 3: Architecture of the first prototype.

4.1 Prototype of the Master

Figure 4 shows a picture of the very first prototype of

the master.

Figure 4: First prototype of the master.

The customer stands in front of the wooden

console. A 60cm × 40cm graphical user interface

(GUI) is projected through the console from behind

by a video-projector. An example of what the

customer sees is showed in Figure 5.

The wooden console appears “transparent” to the

customer thanks to a particular assembly process. The

console is made of a board of transparent PMMA

(polymethyl methacrylate), covered by a 0,6mm layer

of wood. The customer interacts with the wooden

board as if it were a tactile surface. A schematic view

of the master is shown on Figure 6.

Design and Development of the First Prototype of a Social, Intelligent and Connected Help Desk

123

Figure 5: Picture showing the transparent wooden table.

Figure 6: Schematic view of the master.

As previously mentioned in section 3.2,

OpenPose analyses in real-time the video stream

provided by the camera, in order to detect the position

of customer joints. Figure 7 shows an image

processed by OpenPose.

OpenPose is able to detect the hand position, but

the computing requires too many resources and

system performance drastically slows down. That’s

why our software uses the position of the customer

wrist to interact with the GUI. “Selectable” areas are

defined for each screen. Thus the position of the wrist

with respect to these areas, can trigger item selection.

In this prototype, the customer has the possibility to

select a language between French and English and a

service, between wine discovery and restaurant

exploration.

Figure 7: Picture showing the result of OpenPose

processing on the video image.

In order to enhance the CISC interaction with the

customer, a dynamic wall is placed behind the master.

The aim of the wall is to draw the customer attention

and to provide original interaction and help. In order

to draw different patterns, the wall is composed of

thirty cubes of 10cm side, arranged in a 6x5 matrix.

Each cube can move back and forth of a tuneable

depth. Cubes are made of corian

, a composite white

material that can diffuse light. Six programmable

RGB LEDs WS2812B are therefore fixed in the

inside of the cube. Each cube is fixed on a ball bearing

runner and is independently driven by a servo motor

MG945 from TowerPro. A servo driver and a

programmable electronics allow the simultaneous

and independent control of each cube. Figure 8 shows

the first prototype of cube and the final CAD design

of the optimized cube.

Figure 8: First prototype of moving cube on the left,

manufactured by the professional woodworker of the

project. Final CAD design of the cube on the right.

The software architecture of the master, based on

ROS middleware, is showed in Figure 9. The main

role of the CISC node is to activate an Agent node for

each customer in the interaction area. The agent is the

entity that is associated to the customer. It stores

customer preferences, that are saved and transmitted

to terminals whose services have been booked.

Therefore, a communication node is responsible to

send and receive data from the master to the terminals

CHIRA 2019 - 3rd International Conference on Computer-Human Interaction Research and Applications

124

and vice-versa. Data consist of customer choices (e.g.

types of wine or restaurant, particular plate) and time

spent on each choice.

Figure 9: ROS architecture (nodes and topics) of the master.

4.2 Prototype of the Terminals

Two terminals are under development. All terminals

have a similar geometry, dimensions and architecture.

They are activated by the badge and they are

connected to the master by Wi-Fi. The activation of

the terminal is highlighted by a ring of programmable

RGB LEDs. We use a Raspberry Pi 3 B+ electronic

board to control all the input/output devices. A first

terminal presents the wine production of Occitanie, a

region of France. A second terminal presents the

restaurants of Toulouse, a town in France. Only the

wine terminal is presented in details.

Figure 10: The wine terminal designed by the professional

designer of the project.

Figure 10 shows an image of the wine terminal while

Figure 11 shows the architecture of the wine terminal.

The customer can select a department of Occitanie by

touching the corresponding area on a map engraved

(by a laser printer) on the wooden board. A wooden

bottle, integrating a 4,3” touch screen, communicates

with the terminal by Bluetooth. It allows the

visualisation of different information about wines,

corresponding to the selected areas. The bottle is

autonomous in energy and can by grasped by the

customer. Capacitive technology is used to make the

wood tactile (map). On the bottom of the board,

electrodes are painted with conductive ELECTRIC

PAINT™ from BARE Conductive. The customer

hand is the second electrode of the capacitance,

whose variation are measured by the CAP1188

capacitive touch sensor. Sensitivity and threshold

tuning for each sensor enables to reach a stable

configuration. Sensor input determines the activation

of the bottle.

Figure 11: Architecture of the wine terminal.

5 FIRST RESULTS

The results of this section address technical issues of

the design and the implementation of the wooden-

based interfaces previously presented. The complete

CISC prototype is still at a development stage and it

is too early to perform UX tests.

First results concern the tactile wooden surface at

the master. A preliminary testing campaign has been

performed. 15 people, male and female aged from 20

to 50, interacted with the master for 2 minutes. A GUI

composed by several sub-menus, has been

programmed for these tests. This campaign pointed

out an instability in wrist detection, leading to

troubles in obtaining a stable area selection. The

implementation of a Kalman filter solved this issue.

Design and Development of the First Prototype of a Social, Intelligent and Connected Help Desk

125

Figure 12: Adjustment of the right hand position. A similar

procedure is applied to the left hand. Alpha (α) is the angle

between the reference line and the wrist-O line.

Tests also revealed another problem. The users

found particularly difficult the selection of interactive

areas on the top corner of the screen. Because we use

the wrist position to compute the hand position (see

section 4.1), we have to adjust the computed

coordinates of the hand (see Figure 12) on the

reference frame of the GUI.

Table 1: Materials tested with the pattern.

Material

Thickness in mm

Wood veneer pressed wood

Stratified veneer pressed wood

3-ply plywood picea

Glued laminated hevea

Concerning the tactile surface of the wine

terminal, a set of tests have been performed on

different panels (see Table 1), in order to select the

most suitable material for the integration of capacitive

sensors.

Figure 13: On the left, the 61 spots tested for each material.

On the right a picture of the electrodes positioned on the

bottom of each panel.

On the bottom of each panel, we placed a pattern,

made of 9 square painted electrodes of 5cm side,

separated by a space of 2cm (interstice) and arranged

in a 3x3 matrix. Each electrode is connected to the

capacitive sensor (see Section 4.2).

We asked the test users to touch the wood plate on

points aligned to the spots, labelled with a cross on

Figure 13. For each electrode, 5 spots have been

tested, as well as spots in interstices between

electrodes. A contact is detected if the sensor value

exceeds the sensor threshold. Normally, when

touching spots between electrodes (in interstices),

none of the adjacent sensors should detect a contact.

Figure 14: Results obtained for the 4 types of material. A

spot on the electrode in black colour means that the sensor

didn’t sense the contact (unexpected result); grey colour

means the opposite (expected result). In a similar way, a

spot on the interstice in black colour means that none of the

adjacent sensors detected the contact (expected result); grey

colour means that one of the sensors immediately adjacent

to the interstice detected the contact (unexpected result).

Figure 14 shows the results obtained with fixed

sensor sensitivity and threshold. For the tests, the

sensitivity is fixed to 0,5%, which corresponds to a

maximum capacitance variation (ΔC) detection of

150fF from a 30pF base capacitance. The threshold is

an integer value between 0 and 127 and is fixed to

120, that corresponds to a threshold of about 140fF.

The results show that sensor response can be adjusted

by varying sensor sensitivity and threshold.

Nevertheless, for a low value of sensitivity, as the one

used for the tests of Figure 14, for 3 materials over 4,

it is hard to avoid the activation of a sensor when

touching an interstice. Therefore, solid hevea has

been chosen for the final prototype.

Figure 15 shows how results vary by modifying

sensitivity and threshold for the same panel.

CHIRA 2019 - 3rd International Conference on Computer-Human Interaction Research and Applications

126

Figure 15: Results obtained with 3-ply plywood picea.

6 CONCLUSIONS

In this paper we have presented an innovative

architecture of help desk for tourism environment. A

first prototype including a master and a terminal has

been built. First results show the consistency of the

wood-based human-machine interface. Extensive

tests will be now performed to improve software and

hardware architectures in order to fully satisfy the

CISC concept: to integrate available and low cost

technologies in an original design and architecture, in

order to provide the staff of a touristic business or

service with a powerful working tool and to provide

the customer with a unique user experience. CISC is

designed as a service provider, meant to be

intelligent, social and connected. Finally, its modular

design and architecture makes it adaptable to many

environments, applications and budgets. User studies

and data collection will then be conducted.

ACKNOWLEDGEMENTS

The CISC project is financed by FEDER/FSE funds

in the framework of the Readynov call of the

Occitanie region, France. The non-academic all

French partners of the project are: Aranda-Mas, a

woodworker manufacturer, Opteam-Design, an

architect’s office, Beenetic, a company specialised in

IoT technologies.

REFERENCES

Allabarton R. (2018). A complete, actionable guide to the

UX. Retrieved from: https://careerfoundry.com/en/

blog/ux-design/the-ux-design-process-an-actionable-

guide-to-your-first-job-in-ux/ (Accessed 1 May 2019).

Bergweiler S., Deru M. and Port D. (2010, November).

Integrating a Multitouch Kiosk System with Mobile

Devices and Multimodal Interaction. ACM

International Conference on Interactive Tabletops and

Surfaces, Saarbrücken, Germany.

Bhatta S.R., Tiippana K., Vahtikari K., Hughes M. and

Kyttä M. (2017, March 13). Sensory and Emotional

Perception of Wooden Surfaces through Fingertip

Touch, Frontiers. Frontiers in Psychology. doi:

10.3389/fpsyg.2017.00367

Campos, P., Ehrenberg, N. and Campos, M. (2018).

Designing Interactions with Furniture. Towards Multi-

sensorial Interaction Design Processes for Interactive

Furniture. Proceedings of the 20

International

Conference on Enterprise Information Systems, (Vol.2

pp. 527-532).

Cao, Z., Hidalgo, G., Simon, T., Wei, S.-E. and Sheikh, Y.

(2017, July). OpenPose: Realtime Multi-person 2D

Pose Estimation Using Part Affinity Fields. IEEE

Conference on Computer Vision and Pattern

Recognition, Honolulu, HI, USA.

Cassell J., Stocky T., Bickmore T., Gao Y., Nakano Y.,

Ryokai K., Tversky D., Vaucelle C. and Vilhjálmsson

H. (2002, February). MACK: Media lab Autonomous

Conversational Kiosk. Proceedings of IMAGINA’02,

Monte Carlo.

Fossdal M. and Berg A. (2016, September). The

Relationship between User and Product: Durable

Design through Personalisation. International

Conference on Engineering and Product Design

Education, Aalborg, Denmark.

Gómez-Carmona O., Casado-Mansilla D. and López-de-

Ipiña D. (2018, December). Multifunctional Interactive

Furniture for Smart Cities. Proceedings of 12

International Conference on Ubiquitous Computing

and Ambient Intelligence, Punta Cana, Dominican

Republic.

Ikei H., Song C. and Miyazaki Y. (2017, February).

Physiological effects of wood on humans: a review.

Journal of Wood Science (Vol. 63(1), pp. 1-23).

Niculescu A.I., Yeo K.H. and Banchs R.E., 2016.

Designing MUSE: A Multimodal User Experience for

a Shopping Mall Kiosk. In Proceedings of HAI '16.

Omojola O., Post E. R.,Hancher M.D., Maguire Y., Pappu

R., Schoner B., Russo P.R., Fletcher R. and

Gershenfeld N. (2000). An installation of interactive

furniture. IBM Systems Journal, (Vol. 39(3,4)). doi:

10.1147/sj.393.0861

Rogers Y., Sharp H. and Preece J. (2011). Interaction

Design – Beyond Human-Computer Interaction: John

Wiley & Sons 3

edition.

Schafer G.J., Fullerton S.K., Walker I., Vijaykumar A. and

Green K.E. (2018). Words Become Worlds: the Lit

Room, a Literacy Support Tool at Room-Scale. ACM

Proceedings of the Designing Interactive Systems

Conference, (pp. 511-512).

Design and Development of the First Prototype of a Social, Intelligent and Connected Help Desk

127