Development of an Immersive Simulation Platform to Study

Interactions between Automated Vehicles and Pedestrians

Lucie Lévêque

, Thierry Bellet

, Jean-Charles Bornard

, Jonathan Deniel

, Maud Ranchet

Estelle De Baere

and Bertrand Richard

Laboratory of Ergonomics and Cognitive Sciences Applied to Transport, Université Gustave Eiffel, Lyon, France

Engineering Systems International (ESI) Group, CIVITEC, Lyon, France

Keywords: Vehicle/Pedestrian Interactions, Automated Vehicle (AV), Simulation, Human-centred Design.

Abstract: The importance of informal communication between manual vehicles drivers and pedestrians in order to

prevent misinterpretation, and thus accidents, in road-crossing situations has been widely shown in the

literature. Such crucial communication consequently raises the issue of the introduction of automated vehicles

(AVs) on the roads, in which case the status of the driver becomes less obvious. In this paper, we present a

novel simulation platform, the V-HCD, allowing the conduct of immersive experimentations, both from the

pedestrian’s and the driver’s point of view. This platform will be used to study the acceptance of the automated

vehicle for the European SUaaVE project, and further to support the human-centred design of a future

empathic AV.

1 INTRODUCTION

With the rapid development of new technologies and

automation, the introduction of automated vehicles

(AVs), or connected automated vehicles (CAVs), i.e.,

cars without active drivers, offers a strong potential to

increase both traffic safety and accessibility. In fact,

in the near future, the traffic system will be shared

between fully automated vehicles, partially

automated vehicles, and manually driven vehicles

(Litman, 2020). However, it should be noted that

automated vehicles (i.e., levels 3 to 5 of driving

automation (SAE, 2018)) will not only coexist with

more conventional vehicles (i.e., levels 0 to 2 of

driving automation), but also with vulnerable road

users, including pedestrians. Generally, pedestrians

are considered an important indicator of a society’s

health and safety. With a view to contribute to a safe

traffic system, and consequently to increase the

acceptance of AVs, one key challenge is therefore to

investigate how the latter interact with pedestrians.

To achieve safe interactions, manual vehicle

drivers and pedestrians need to share their

understanding and awareness of the traffic situation

(Endsley, 1995; Bellet et al., 2009). Otherwise,

critical conflicts may occur; 21% of fatal road traffic

accidents happen to involve pedestrians (WHO,

2015). It is crucial to account that misinterpretation of

others’ intentions is one of the main causes of

accidents involving pedestrians (Habibovic et al.,

2012). This is particularly true in the case of road

crossing decision-making, and especially when

priority rules are unclear (e.g., absence of zebra

crossing). In such a context, pedestrians and manual

drivers frequently interact using non-verbal

communication to clarify their intentions. Several

studies have shown the importance of this informal

communication in the literature.

For instance, Schmidt et al. found that pedestrians

who want to cross a street tend to look at the

approaching vehicle to get acknowledgement from

the driver; if the driver returns their eye contact,

pedestrians assume that they have been seen and that

they have achieved a mutual understanding (Schmidt

et al., 2009). Similarly, Sucha et al. found that

pedestrian’s decision to cross, as well as their feeling

of safety, are directly impacted by various signals

provided by the driver, like eye contact, postures,

waving hand, or flashing lights (Sucha et al., 2017).

Such conclusions were also drawn by Rasouli et al.

who showed that the most prominent signal to

transmit pedestrians’ crossing intention is looking, or

at least glancing, towards oncoming traffic (Rasouli

et al., 2017). Finally, in their study, Schneemann et al.

found that, when pedestrians interact with vehicles at

low speed, they tend to rely on eye contact with the

driver; whereas, at faster speed, they generally base

Lévêque, L., Bellet, T., Bornard, J., Deniel, J., Ranchet, M., De Baere, E. and Richard, B.

Development of an Immersive Simulation Platform to Study Interactions between Automated Vehicles and Pedestrians.

DOI: 10.5220/0010125002490254

In Proceedings of the 4th International Conference on Computer-Human Interaction Research and Applications (CHIRA 2020), pages 249-254

ISBN: 978-989-758-480-0

249

their decisions on the dynamics of the vehicle

(Schneemann et al., 2017).

To summarise, all these studies clearly indicate that

an active communication between pedestrians and

manual vehicle drivers is a crucial element to manage

situational risks, to support pedestrians’ decision-

making, and to increase their safety while crossing

roads. A key concern regarding the introduction of

automated vehicles public roads is therefore due to

the changing status of the drivers. Indeed, AVs may

negatively impact interactions with pedestrians as

they will not be able to rely on cues from drivers’

behaviours anymore, potentially leading to

uncertainty and mistrust (Vissers et al., 2017).

Malmsten Lundgren et al. suggested that the

introduction of automated vehicles in the urban

context may lead to a notable change in how

pedestrians experience AVs compared to

conventional vehicles (Malmsten Lundgren et al.,

2016). In their study, pedestrians rated eye contact

with a driver as promoting safe interaction; whereas

apparent driver distraction in an AV (e.g., phoning or

reading the newspaper) tended to increase

pedestrians’ anxiety.

2 RESEARCH OBJECTIVES

Regarding automated vehicles, it should be noted that

one key challenge is not only to study the

acceptability (i.e., before use), but also the acceptance

(i.e., after use) (Schade et al., 2003; Distler et al.,

2018). However, automated and/or autonomous

vehicles are not commonly traveling on European

roads yet. Therefore, the only way to be able to study

the aforesaid acceptance is by using new generation

immersive simulation tools, which allow users to plan

for the future thanks to virtual reality (Kyriadikis et

al., 2019). By developing such immersive simulation

environments, it makes it possible to invite real

humans to practically experience future technologies

and situations. Investigating the adequacy of future

systems to end users’ needs, exploring potential risks,

and evaluating the acceptance then become

conceivable.

The in-depth study of how pedestrians and

automated vehicles interact with each other is a

crucial issue for the SUaaVE (SUpporting acceptance

of automated VEhicle) project. In this context,

developing an immersive platform gathering such

virtual reality tools would allow the people of today

to experience the AVs of tomorrow. In order to reach

this goal, Université Gustave Eiffel (ex-IFSTTAR)

and ESI/CIVITEC decided to create such an

immersive simulation platform for the SUaaVE

project, through a pre-existing tool.

3 DESIGN AND DEVELOPMENT

OF A V-HCD PLATFORM

With a view to better understand and integrate users’

needs in the design of innovative advanced driving

aid systems (ADAS), IFSTTAR and ESI Group

jointly developed a virtual human-centred design

platform; the V-HCD (Bellet et al., 2012, 2018). As a

virtual simulation toolbox, the V-HCD is able to

handle human-based simulation, that is to say based

on a virtual driver model, to virtually assess accident

risks as well as the potential benefits of future ADAS.

The V-HCD integrative platform is made of two

main components: a virtual driver, and a virtual

prototyping platform. The virtual driver is based on

the cognitive model COSMODRIVE, i.e., COgnitive

Simulation Model of the DRIVEr (Bellet et al., 2009).

The virtual prototyping platform, named ESI Pro-

SiVIC

integrates simulated infrastructures, road

users, vehicle dynamics, and multi-technology

perception sensors (Gruyer et al., 2006).

In a recent study, Bellet et al. (Bellet et al., 2019)

used this platform for the human-centred design of a

driver monitoring system, with a view to identify

risks of collision caused by visual distraction during

driving, and in charge to manage human-machine

interactions in real time. Figure 1 represents an

example of simulation with the COSMODRIVE

model implemented in the V-HCD, to simulate the

effects of visual distraction on the driver’s situation

awareness.

In this previous project, the V-HCD was used in

the initial stages of the design process, during which

there was no prototype of the future system, and

therefore no experimentation with real humans nor

user testing was possible. In such an early stage,

making use of a user model, such as COSMODRIVE,

can enable a better comprehension of the risks due to

visual distraction depending on the situational

context. COSMODRIVE indeed allowed the

simulation of various driving scenarios with different

levels of driver distraction in order to identify the

most critical situations, as illustrated in Figure 2.

These critical scenarios were then used as ‘reference

use cases’ (Bellet et al., 2019) to consequently specify

functionality to embed in future ADAS with a view

to prevent accidents.

SUaaVE 2020 - Special Session on Reliable Estimation of Passenger Emotional State in Autonomous Vehicles

250

Figure 1: Illustration of the simulation of drivers’ visual distraction effects with the V-HCD.

Figure 2: Illustration of an example of simulated accident

risks due to visual distraction with the V-HCD platform.

Beyond this early use, the V-HCD can also be

used at more advanced stages of the design process.

Indeed, as soon as models and/or real or virtual

prototypes of the future ADAS are available, the V-

HCD can be used as an integrated simulator of: (1)

the ADAS, (2) the road environment, and (3) the

vehicle. Then, real humans can sit in the vehicle to

put themselves in the shoes of future users of this

ADAS, and consequently have a realistic experience

of this future technology before it becomes available

on real cars.

In the framework of the SUaaVE project, the

objective was to make changes to the original V-HCD

platform, in order to focus on the study of AV-

pedestrians interactions and, more precisely, on the

acceptance of such automated vehicles. As introduced

in the previous section, the latter are not traveling on

the roads yet; therefore, the only way to investigate

acceptance is through the use of virtual immersion.

It is with this in mind that the new version of the

V-HCD platform was designed and developed for the

SUaaVE project, i.e.: to allow users today to plunge

into an immersive experience of interaction with an

autonomous vehicle, either from the driver/passenger

point of view interacting with a virtual pedestrian, or

from the pedestrian point of view, willing to cross in

front of an AV.

Figure 3 describes this perspective by presenting

a short scenario where a pedestrian is crossing the

street while an automated vehicle is approaching.

More precisely, Figure 3 (view 1) shows the

designer’s overview, where the parameters (e.g., cars

colours, distances, speeds, accelerations) of each

object can be tuned.

Figure 3 (view 2) represents a possible

perspective of the aforementioned scenario, where a

human subject becomes the pedestrian. Thanks to a

virtual reality (VR) headset, they can turn their head

to check for oncoming traffic and take the pedestrian

crossing. The avatar settled in the oncoming AV can

then be positioned at different places (i.e., front or

rear seats), and participate in various activities (e.g.,

driving carefully, sleeping on the steering wheel,

talking to a rear passenger), as illustrated in Figure 4.

The objective of these diverse situations is to analyse

whether, and to what extent, the decision to cross is

modified.

Finally, Figure 3 (view 3) corresponds to a more

traditional use of simulation, where a participant

takes the place of an occupant of the AV. In such a

case, they experience the situation either through the

use of a traditional driving simulator cabin, or thanks

to a VR headset creating a virtual cockpit. The subject

can also multi-task while driving (e.g., reading a

book, playing a game). The subject can hence

experience diverse AV behavirous, and later express

their feelings about each of them. Furthermore,

different demeanours can be associated with the

pedestrian (e.g., crossing quickly, changing their

mind and stepping back).

When implemented simultaneously thanks to the

V-HCD, both perspectives (i.e., as a pedestrian and as

an AV occupant (driver or passenger), respectively)

can enable two participants to interact with each other

in a simulated world. In this instance, the first

volunteer sits in the driving simulator, while the

second one puts themselves in the position of the

Development of an Immersive Simulation Platform to Study Interactions between Automated Vehicles and Pedestrians

251

Figure 3: Illustration of an example of scenario implemented on the V-HCD platform to study the interactions between a

pedestrian and an AV with a more or less attentive driver.

Figure 4: Illustration of examples of driving scenarios implemented on the V-HCD platform to study the interactions between

a pedestrian and an AV.

pedestrian by means of the VR headset. Both

participants therefore experience the same situation at

the same time, but from a different point of view.

Each of them can witness the other’s behaviour, such

as the potential distraction of the AV occupant, or the

possible hesitation of the pedestrian before crossing.

The V-HCD platform thereby becomes a set of

tools concomitantly integrating automated vehicles

(AVs), advanced driving aid systems (ADAS), and

different ways to immerse current road users in the

future. The AVs are customisable with tailored

behaviours originating from cognitive simulation,

allowing the fine-tuning of diverse situations while

maintaining necessary realism. Simulating ADAS in

the immersive experience of road users can further

help with the integration of future systems. To

summarise, as jointly developed by Université

Gustave Eiffel (ex-IFSTTAR) and ESI/CIVITEC, the

V-HCD allows the testing of AVs and ADAS, as well

as of their acceptance by the future end-users.

4 USE OF THE V-HCD

PLATFORM FOR SUaaVE

In order to study how interactions between

pedestrians and automated vehicles may look like in

the future, and how these interactions may be affected

SUaaVE 2020 - Special Session on Reliable Estimation of Passenger Emotional State in Autonomous Vehicles

252

by the AV behaviour, Université Gustave Eiffel will

implement a two-phase experiment for the SUaaVE

project. The first phase will focus on the AV

passenger/driver’s point of view, whereas the second

phase will be dedicated to the pedestrian’s point of

view.

For the first phase of the experiment, i.e., from the

driver/passenger’s point of view, participants will

experience how the AV reacts when facing diverse

pedestrians’ road crossing behaviours (i.e., more or

less expected and/or critical). For this phase, two

different conditions will be considered: with, or

without zebra crossing. Several AV behaviours are

implemented on the V-HCD to interact with the

pedestrian; they will be fully managed and performed

by vehicle automation. During each of these

alternatives, participants will have the opportunity to

use the horn to warn the pedestrian. This way, it will

be possible to measure if, and when, they feel that the

interaction is becoming too critical, and potentially

unacceptable.

Participants will afterwards be asked to

assess the situational criticality and the reaction of the

AV according to the pedestrian’s decisions and

behaviours. After having experienced all the

scenarios, participants will be invited to a semi-

structured interview to express their suggestions on

how an “empathic” automated vehicle should react

when interacting with pedestrians, and/or should

inform the latter about its decisions and reactions. At

this level, the objective of these final questions will

be to collect useful information for the future user-

centric design of SUaaVE’s empathic automated

vehicle, namely ALFRED (Automation Level Four

and Reliable Empathic Driver).

The second phase of the experiment will be

conducted thanks to a simulated environment using

virtual reality (VR). Virtual reality indeed allows an

immersive, safe, and controlled study of the

interactions between a pedestrian and an AV

approaching at different speeds, and with distinct

braking behaviours. The second phase of the

experiment will focus on the pedestrian willingness

or unwillingness to cross the street in front of an

automated vehicle, in the absence of zebra crossing,

where priority rules are unclear. More precisely, the

emphasis will be made on the pedestrian’s perceived

safety and decision to cross, or not, the road, when

interacting with an AV. For this test, participants will

be located on a sidewalk, facing a continuous flow of

approaching vehicles. First, a randomised number of

vehicles, separated with short gaps, will travel

without stopping; then, an automated vehicle will

appear on the scene. Different behaviours will be

implemented in the AV (i.e., in terms of dynamics

and ways to stop). The AV occupant, simulated by an

avatar, may have different on-board activities (e.g.,

phoning or discussing with another passenger) or

attentive/distracted status, as illustrated in Figure 4.

Participants will be asked to use a joystick to assess

the safety versus dangerousness of crossing the road

from the pedestrian’s point of view. Depending on the

estimated safety level; the more they will estimate the

situation as safe, the further they will have to push the

joystick forward. On the contrary, the more the

crossing will be assessed as dangerous, the further

they will have to pull the joystick backwards.

Intending to cross a road being an active decision,

keeping the joystick in a neutral position during the

whole scenario will correspond to an intention not to

cross from the beginning. It will thus be possible to

collect the participants’ risk assessment in a dynamic

way throughout the approaching phase of the AV, and

this, whether or not the AV stops.

5 CONCLUSION AND

PERSPECTIVES

In this paper, we presented a new immersive

simulation platform, the V-HCD. This platform was

designed to allow users of today to “anticipate the

future” by virtually plunging into a simulated, yet

realistic, situation of interaction with automated

vehicles. This happens to be particularly useful to

study the acceptance (after a first use of a technology

thanks to virtual reality), and not only the a priori

acceptability (i.e., without any practical experience),

as well as the relevance of virtual reality to support a

cross simulation.

For the SUaaVE (SUpporting acceptance of

automated VEhicles) project, this platform will be

used in two different, notwithstanding

complementary stages. Indeed, a single participant

will be able to experience the situation both from the

pedestrian’s point of view, as well as from the

automated vehicle (AV) occupant’s point of view.

Furthermore, as a result of the efforts performed and

using the experience learnt from this project, it is also

expected to progress towards an interactive multi-

users V-HCD supported by cross-simulation: where

several users can experience a given situation and

interact in real time, some of them taking the position

of pedestrians, and other ones the roles of AV

occupants (i.e., driver or passenger). To conclude,

this new approach of virtual cross-simulation opens

the gate to different types and multiple synchronised

simulations, taking into account humans as end-users

and their different mobilities.

Development of an Immersive Simulation Platform to Study Interactions between Automated Vehicles and Pedestrians

253

ACKNOWLEDGEMENTS

This project has received funding from the European

Union’s Horizon 2020 research and innovation

programme under grant agreement No 814999.

REFERENCES

Bellet, T., Bailly-Asuni, B., Mayenobe, P., Banet, A., 2009.

A theoretical and methodological framework for

studying and modelling drivers’ mental representations.

In Safety Science, vol. 47, pp. 1205-1221.

Bellet, T., Mayenobe, P., Bornard, J-C., Gruyer, D.,

Claverie, B., 2012. A computational model of the car

driver interfaced with a simulation platform for future

Virtual Human Centred Design applications: COSMO-

SIVIC. In Engineering Applications of Artificial

Intelligence, vol. 25, pp. 1488-1504.

Bellet, T., Richard, B., Laverdure, S., Bornard, J-C., 2018.

Use of a cognitive simulation model of the driver to

support the Virtual Human Centred Design (V-HCD) of

ADAS and automated vehicles. Graz Symposium

Virtual Vehicle, Graz, Austria.

Bellet, T., Bornard, J-C., Deniel, J., Richard, B., 2019.

Driver modelling and simulation to support the Virtual

Human Centred Design of future aids. Human Systems

Integration Conference, Biarritz, France.

Distler, V., Lallemand, C., Bellet, T., 2018. Acceptability

and acceptance of autonomous mobility on demand:

The impact of an immersive experience. Conference on

Human Factors in Computing Systems, Montréal,

Canada.

Endsley, M., 1995. Toward a theory of situation awareness

in dynamic systems. In Human Factors, vol. 37, pp. 32-

64.

Gruyer, D., Royère, C., Du Lac, N., Michel, G., Blosseville

J-M., 2006. SiVIC and RTMaps, interconnected

platforms for the conception and the evaluation of

driving assistance systems. World Congress and

Exhibition on Intelligent Transportation Systems and

Services, London, United Kingdom.

Kyriakidis, M., De Winter, J., Stanton, N., Bellet, T., Van

Arem, B., Brookhuis, K., Martens,

M., Bengler, K.,

Andersson, J., Merat, N., Reed, N., Flament, M.,

Hagenzieker, M., Happee, R, 2019. A human factors

perspective on automated driving. In Theoretical Issues

in Ergonomics Science, pp. 1-27.

Habibovic, A., Davidsson, J., 2012. Causation mechanisms

in car-to-vulnerable road users crashes: Implications for

active safety systems. In Accident Analysis and

Prevention, vol. 49, pp. 493-500.

Litman, T., 2013-2020. Autonomous vehicle

implementation predictions. Victoria Transport Policy

Institute.

Malmsten Lundgren, V., Habibovic, A., Andersson, J.,

Lagrström, T., Nilsson, M., Sirrka, A., 2016. Will there

be new communication needs when introducing

automated vehicles to the urban context? In Advances

in Human Aspects of Transportation, pp. 485-497.

Rasouli, A., Kotseruba, I., Tsotsos, J., 2017. Agreeing to

cross: How drivers and pedestrians communicate. In

IEEE Intelligent Symposium (IV), Washington, USA.

Schade, J., Schlag, B., 2003. Acceptability of urban

transport pricing strategies. In Transportation Research

Part F: Traffic Psychology and Behaviour, vol. 5, pp.

45-61.

Schmidt, S., Färber, B., 2009. Pedestrians at the kerb:

Recognising the action intentions of humans. In

Transportation Research Part F: Traffic Psychology

and Behaviour, vol. 12, pp. 300-310.

Schneemann, F., Gohl, I., 2017. Analyzing driver-

pedestrian interaction at crosswalks: A contribution to

autonomous driving in urban environments. In IEEE

Intelligent Vehicles Symposium (IV), Washington,

USA.

Society of Automotive Engineers (SAE), 2018. Taxonomy

and definitions for terms related to driving automation

systems for on-road motor vehicles. J3016-201806.

Sucha, M., Dostal, D., Risser, R., 2017. Pedestrian-driver

communication and decision strategies at marked

crossings. In Accident Analysis and Prevention, vol.

102, pp. 41-50.

Vissers, L., Van der Kint, S., Van Schagen, I., Hagenzieker,

M., 2017. Safe interaction between cyclists,

pedestrians, and automated vehicles: What do we know

and what do we need to know? Institute for Road Safety

Research, Report R-2016-16.

World Health Organisation (WHO), 2015. Global status

report on road safety.

SUaaVE 2020 - Special Session on Reliable Estimation of Passenger Emotional State in Autonomous Vehicles

254