Estimating Driver Unawareness of Pedestrian based on Visual

Behaviors and Driving Behaviors

Minh Tien Phan, Indira Thouvenin,Vincent Fremont and Véronique Cherfaoui

Heudiasyc Laboratory, University of Technology of Compiègne, Compiègne, France

1 STAGE OF THE RESEARCH

Every year, more than 1.24 million people die in

traffic accidents around the world, half of which are

among pedestrians and cyclists, and 22 percent of

which are children (WHO, 2013). Thus, preventing

pedestrians from motor accidents is an important

problem in achieving a safe and secure society. A lot

of research in the advanced driver assistance is able

to detect the pedestrian with the in-vehicle sensors

and inform the driver of their presence. However,

most of these alert systems do not adapt to the

driver, they become distracting, annoying the driver

and will be often ignored or deactivated. Therefore,

taking into account the driver’s state and

understanding fully the situation are the challenges

for these systems.

This study is the first step of a project named

MINARDA (Mobile and Interactive Augmented

Reality for Driving Assistance). This system is

strongly connected to the user. It aims to use a

suitable visualization metaphor in augmented reality

that allows the driver to avoid dangerous situations.

Particularly, we focus on the situation in which the

pedestrian is in danger.

This paper will be structured as follows. In

sections 2 and 3, we underline our objectives of

research and define the problem. In section 4, we

review some related works. In Section 5, we

describe the proposed methodology to treat this

problem. Section 6 describes our expected outcome,

conclusion and perspectives.

2 OUTLINE OF OBJECTIVES

As mentioned above, the MINARDA system needs

to fully understand the situation to be able to provide

the appropriate cues to the user. In the context of

pedestrian safety, our first objective is to find out

what the features that characterize the driver’s

awareness or unawareness of pedestrian are. Once

we can identify these features, our second objective

is to build a model of the awareness and

unawareness a driver has of pedestrians appearing

on the road ahead. At last, we will envisage

estimating the different levels of danger to

pedestrian. The more precisely the danger levels are

classified, the more suitably the assistance cues can

adapt to the driver.

3 RESEARCH PROBLEM

In a previous work, Engel and Curio (2011, 2012),

Wakayama, (2012) proposed to estimate how easy it

is for the driver to see and notice pedestrians. The

studies included the estimate of driver’s visual

attention and pedestrian detectability in an image. In

this study, we aim to provide a new and more direct

approach. It can be assumed that the driver’s

awareness or unawareness of pedestrian could be

estimated by observing his visual behaviors and

driving behaviors. More precisely, our hypothesis is

whenever the driver is aware of a pedestrian ahead,

he has to direct his gaze to pedestrian and does some

reactions on vehicle, (brake, deceleration, wheel

angle change, etc.) what is not in the case of

unawareness.

This preliminary suggestion leads to two related

questions:

1. Is there any difference in visual behaviors and

driving behaviors between two states driver’s

awareness of and unawareness of something on road

(pedestrian in this case)?

2. Is it possible to extract and use those patterns

to help identify or learn information about the

danger to pedestrian?

In order to answer these questions, we have to

envisage different experiments and consider

different methods to measure visual behaviors and

driving behaviors. Because of their reliability and

flexibility, in this work, we use a vision-based

method to observe the visual behaviors and various

in-vehicle sensors to measure the driving behaviors.

Tien Phan M., Thouvenin I., Fremont V. and Cherfaoui V..

Estimating Driver Unawareness of Pedestrian based on Visual Behaviors and Driving Behaviors.

 2014 SCITEPRESS (Science and Technology Publications, Lda.)

4 STATE OF THE ART

In the state of the art includes the related research

that helps to understand the danger to pedestrian,

particularly those are related to the driver. Thus, we

talk about the problem of pedestrian detectability,

we review in detail the studies of driver inattention

detection, and the studies which are directly related

to the driver’s awareness of the pedestrian. We will

then identify the strength and weakness of each

method that should be considered in our works.

4.1 Pedestrian Detectability

The studies on the detectability of the pedestrian aim

to estimate how easy it is for the driver to see and

notice pedestrians. In the work of Engel and Curio

(2011), the authors did the experiments to measure

the time to recognizing the pedestrian on image.

Then, they determined the features of pedestrians

that correlated with this time. Thus, the detectability

of pedestrian was modeled. The features of

pedestrian on image could be for example, the

position of pedestrian, size, color, light, etc. This

study was also extended to the dynamic scene

concept. Actually, instead of analyzing the

pedestrian’s detectability on images, it is possible to

do the experiments on the sequence video (Engel

and Curio, 2012). Another extension was proposed

by Wakayama (2012) who considered the motion of

pedestrian to be one of the features that characterize

the pedestrian detectability. However, as before,

because the fact that the pedestrians can be seen

easily does not mean that the driver is cognitively

aware of them (Fukagawa, Yamada, 2013).

4.2 Driver Inattention Detection

The lack of attention by drivers is the biggest single

cause of serious road accidents. A great amount of

work have been done on the inattention state of the

driver. However, the inattention of the driver was

defined in different ways. In this part (4.2), the

inattention means the abnormal states of the driver

such as fatigue, drowsiness or distraction, etc.

Otherwise, in the rest of our research, driver’s

unawareness of pedestrian is considered to be a kind

of inattention based on this definition: “Driver

Inattention means insufficient or no attention to

activities critical for safe driving” (Regan et al,

2011). According to this, unawareness of the

pedestrian means insufficient or no attention to this

pedestrian for safe driving.

The driver’s behaviors and his inattention state

are analyzed and detected under different

techniques, the majority of which are as follows.

4.2.1 Physiological Measures Observation

The physiological signals such as heart rate,

respiration or EEG and are considered to be the most

reliable signals to analyze the driver behaviors and

to detect inattentiveness.

Heart rate is easily determined through

Electrocardiogram (ECG) signal. The mental stress

while driving for example, increases blood pressure,

heart rate and activate the sympathetic nervous

system. (Moriguchi et al., 1992). There are

significance impacts in the heart rate when a driver

is in a state of cognitive inattention (thinking while

driving for example) (Akin et al, 2008).

Moreover, the EEG (Electroencephalogram)

signal has various frequency bands which can

signify different states of the driver. For example,

there is evidence that the increase of the alpha band

(8-13Hz) corresponds to relaxation. Increase of the

delta band (0.5-4Hz) and the theta band (4-8Hz)

correspond to sleep activity and drowsiness which

signify a potential failure of attention and behaviors.

(Östlund, 2004), (Santana, 2002). In another

experiment, twelve types of energy parameters

computed from three bands alpha, beta, and delta

were chosen as the evaluation index of driver fatigue

(Li, 2012)

However, these techniques are obtrusive, since it

is necessary to attach some electrodes on the drivers,

causing annoyance to them. Consequently, it is

difficult to be implemented in the real-life

applications.

4.2.2 Visual Behaviors Observation

Driver behaviors could be somehow observed from

the changes in their facial features like eyes, head

and face. In this context, several methods have been

proposed. This measurement has the advantage of

being unobtrusive, since they could be collected

with remote eye-head trackers. Moreover, they are

quite reliable.

Measurements related to head movement such as

nodding frequency and those related to the eye

region such as eyelid distance changes, eye close

duration, blinking frequency, and the recently

developed parameter PERCLOS (percentage of time

in a minute that the eye is 80% closed) (Lin,2012)

were widely used in research. After extracting these

features, Bergasa (2008) and Mohamad-Hoseyn

Sigari (2013) built a fuzzy model or finite state

VISIGRAPP2014-DoctoralConsortium

machine to estimate the inattentive, distracted or

fatigue state of driver.

Another cue is the size of the pupils. The pupils

is the part of the iris that allows light to enter the

retina. Besides light, the pupil dilates when mental

or cognitive effort is given. It was observed that the

pupil was dilated and the diameter of average pupil

size increased by 15% when the driver was

cognitively distracted (Akin et al, 2008), (Benedetto,

2011).

Gaze behaviors are another cues that can be used

as a metric to find if the concentration of the driver

was on driving or not. It was noticed that, when the

driver was cognitively distracted, glancing at

instruments and mirrors decreased significantly

(Harbluk, 2007). A simple method to detect the

driver’s attention was proposed by Fletcher and

Zelinsky (2009.). They used driver’s gaze vector and

road scene event correlation to estimate if the driver

has seen the event or not. Due to the “looking but

not seeing” problem, this work was not able to

reliably determine if a certain road event (pedestrian

for example) is perceived by the driver. However it

could identify if the driver has not perceived a road

event.

Deeper in this context, Doshi and Trivedi (2009)

provided a result on the observation of the dynamic

of overt visual attentions shifts. They found that

there are various interactions between head and eye

movements that are useful in detecting the driver

distractions, as well as the driver intent. Indeed, their

results validated the differences existing between

goal-oriented and stimulus-oriented gaze shift.

Moreover, this feature could be observed in

dynamics of eye and head movements. They also

investigated the problem of detecting the intent of

the driver in changing lane which was considered to

be a goal-oriented attention shift. (Doshi and

Trivedi, 2012). This result showed that the driver

attentiveness or awareness of something should be

observed through the dynamic of many parameters.

4.2.3 Driving Behaviors Observation

Another method on the driver’s inattention research

is to detect it indirectly through driving behaviors

such as vehicle speed, steering wheel movement,

lateral position, and break or acceleration pedal

states. (Imamura et al, 2008), (Ueno et al, 1994).

The projects conducted by HASTE (Human

Machine and the Safety of Traffic in Europe)

demonstrated that the steering measures were

significantly affected by the visual task, when a

subject had to perform the visual task, the steering

effort was higher than in the baseline condition.

(Östlund et al., 2004).

The IVDRs (In-vehicle Data Recorder) offer

valuable information on a driver’s behavior through

the analysis of automobile-operating information.

Jensen and Wagner (2011) also proposed a

combination of three analysis methods to evaluate

the driver performance: data threshold violations,

phase plane analysis with limits and a recurrence

plot with outlier limits. These methods were based

on the measures of vehicle speed, engine speed,

vehicle latitude and longitude coordinates, and

lateral accelerations.

Although these techniques are not obtrusive, they

are subject to several limitations such as vehicle

type, driver experience, geometric characteristics,

state of road, etc. (Bergasa, 2008 ).

4.3 Driver Unawareness of Pedestrian

To the best of our knowledge, there are no studies

that are directly related to the estimation of the

driver unawareness of pedestrian or other traffic

events. The most closely related works are those of

(Fukagawa, Yamada, 2013) who proposed a

hypothesis that is likely linked to our work. Their

study was based on the driver’s operational data

such as pressure on the accelerator pedal, pressure

on the brake pedal (called acceleration reaction),

steering angle and vehicle behavior data such as

vehicle speed to estimate the driver’s awareness of

pedestrian.

Their hypothesis was that whenever a pedestrian

appears on the road, if the driver noticed it, he had to

do the acceleration reaction somehow. They used the

driving behavior data collected by Research Institute

of Human Engineering for Quality of Life

(Akamatsu et al., 2003). From that, they proposed to

calculate the probability of acceleration reaction

being observed at a distance given that response to

driver’s aware of the pedestrian. This probability

was assumed to be a log-normal distribution. They

also proposed to calculate the probability of

acceleration reaction being observed at a distance

given that not in response to driver’s aware of the

pedestrian and this probability was considered to be

a uniform distribution. Hence, using Bayes theorem,

they calculated the probability that one acceleration

reaction was caused in response to driver’s

awareness of pedestrian.

However, there is a couple of weaknesses in this

study. Firstly, in the data collected in real and actual

road condition, it was supposed that whenever a

pedestrian appeared on road, the driver had to notice

EstimatingDriverUnawarenessofPedestrianbasedonVisualBehaviorsandDrivingBehaviors

them, this is hard to verify. Secondly, the driver

could totally accelerate if he had been aware of the

pedestrians and identify that they were not in danger.

On the other hand, this probability model is not

reliable because of the use of a specific distribution

law. Finally, this study can’t determine if the driver

has not noticed the pedestrian or has been unaware

of them.

5 METHODOLOGY

In this work, we have defined the driver’s

unawareness and awareness of pedestrian. We use

the camera-based system to observe his visual

behavior and the in-vehicle sensors to measure the

driving behaviors. Moreover, in the experiment, we

have discussed two most important issues as follow.

Hence, the methodology of research is determined

5.1 Driving Simulation Vs. Naturalistic

Driving Data Collection

In our work, both driving simulations and real

conditions driving are expected out. Firstly, a

driving simulation is used to collect all information

from different sensors. From that, the feature vectors

representing the driver’s awareness and unawareness

of pedestrian will be extracted. The simulation

environment is useful because we can control the

situation and can propose different scenario that fit

well to our research problem. For example, we can

propose two scenarios. First, a scenario where the

driver is asked to recognize the gender of a

pedestrian while driving. On the other hand, a

scenario of driver’s unawareness in which the driver

is demanded to do a second visual task while

driving.

Naturalistic driving data help to verify the

hypothesis in real conditions. Many experiments

have been conducted in simulated environment and

the results have been discussed. However, in real

driving conditions the result might be drastically

lower as a moving vehicle presents new challenges

like variable lighting, changing background and

vibration, etc. (Arun et al, 2012). The data collected

in real conditions will be extracted and annotated

whenever a pedestrian appears on road. Then, we

will compare the driver visual reaction and

acceleration reaction with the ones that have been

extracted in simulation driving.

We use the help of MINARDA system

embedded on CARMEN platform to collect the

information.

Figure 1: MINARDA system and CARMEN car- Ready

for simulation driving as well as naturalistic driving data

collection.

5.2 Hybrid Measures

Each previously mentioned method get their own

advantages and limitations. Nevertheless, in the

advanced driving assistance systems, the

environment, the vehicle and the driver have to be

considered to be an overall driving system (Trivedi

et al. 2007). Proposed study is to consolidate various

measures which could lead to a good detection

system. Indeed, beside the pedestrian detection

system that provides all properties of pedestrian on

road (position, distance, time to collision, etc.), we

propose to use the visual behaviors, specially the

looking vector which is provided by SMI Glass and

head motion which is provided by the Facelab

System. In the other hand, the driving behaviors

such as the driver acceleration reaction are measured

by the IVDR. These cues will be collected in high

frequency to analyze the driver’s awareness or

unawareness of pedestrian.

Our hypothesis is that, this kind of driver

inattention should be observed through the change of

the looking reaction with respect to a pedestrian or

the change in his driving acceleration. Or it may be

characterized by the interaction of these both

reactions, for example, by a reaction of looking first

and followed by an acceleration reaction. This

hypothesis may be complete the study of Yuuki

Fukugawa (2013) since we are able to observe the

case when the driver is aware of pedestrian and does

no reaction acceleration and the case when the driver

does the acceleration reaction but he is unawareness

of a pedestrian.

6 EXPECTED OUTCOME

At the end of this study, the reliable patterns that

characterize the driver’s states of unawareness and

awareness of pedestrian are expected to be extracted

out. Then, we intend to develop a new learning

algorithm based on these patterns in order to model

this driver’s behavior and the danger level to

VISIGRAPP2014-DoctoralConsortium

pedestrian. The whole algorithm will be

implemented on the MINARDA system.

In this paper, we talked about the danger to

pedestrian. We provided a review on the methods

that help to analyze the driver behaviors and to

detect his inattention state. We propose a novel

approach directly related to pedestrian safety, the

driver’s awareness and unawareness of pedestrian.

This study could be extended to other type of road

events such as traffic light or other vehicles

detection and help us to understand better the

driving context.

REFERENCES

Akamatsu, M.; Sakaguchi, Y. and Okuwa, M. 2003.

“Modeling of Driving Behavior When Approaching an

Intersection Based on Measured Behavioral Data on

An Actual Road”. Proceedings of the Human Factors

and Ergonomics Society Annual Meeting, 47, 1895-

1899.

Akin, M.; Kurt, M.; Sezgin, N. and Bayram, M.

2008.”Estimating vigilance level by using EEG and

EMG signals”. Neural Computing and Applications,

Springer-Verlag, 17, 227-236.

Arun, S.; Sundaraj, K. & Murugappan, M.2012

“Driver inattention detection methods: A review”

Sustainable Utilization and Development in

Engineering and Technology (STUDENT), 2012 IEEE

Conference on, 1-6

Benedetto, S.; Pedrotti, M.; Minin, L.; Baccino, T.; Re, A.

and Montanari, R. 2011. “Driver workload and eye

blink duration”.Transportation Research Part F:

Traffic Psychology and Behaviour , 14, 199 – 208.

Bergasa, L.; Nuevo, M.; Barea, R. and Lopez, E.

Prokhorov, D. (Ed.). 2008. “Visual Monitoring of

Driver Inattention”. Computational Intelligence in

Automotive Applications, Springer Berlin Heidelberg,

132, 19-37.

Doshi, A. and Trivedi, M. 2009.”Head and gaze dynamics

in visual attention and context learning” Computer

Vision and Pattern Recognition Workshops, 2009.

CVPR Workshops 2009. IEEE Computer Society

Conference on, 77-84.

Doshi, A. and Trivedi, M. 2012. “Head and eye gaze

dynamics during visual attention shifts in complex

environments” Journal of Vision, 12

Engel, D. and Curio, C. 2011. “Predicting human perception

performance with machine vision”. Intelligent Vehicles

Symposium (IV), 2011 IEEE, 429-435.

Engel, D. and Curio, C. 2012. “Detectability prediction in

dynamic scenes for enhanced environment perception”

Intelligent Vehicles Symposium (IV), 2012 IEEE, 178-

183.

Fletcher, L. and Zelinsky, A. 2009. “Driver Inattention

Detection based on Eye Gaze-Road Event

Correlation”. Int. J. Rob. Res., Sage Publications, Inc.,

28, 774-801.

Fukagawa, Y. and Yamada, K. 2013. “Estimating Driver

Awareness of Pedestrians from Driving Behavior

based on a Probabilistic Model”. Intelligent Vehicles

Symposium (IV).

Harbluk, J. L.; Noy, Y. I.; Trbovich, P. L. and Eizenman,

M. 2007. “An on-road assessment of cognitive

distraction: Impacts on driver’s visual behavior and

braking performance”. Accident Analysis &

Prevention, 39, 372 – 379.

Imamura, T.; Yamashita, H.; bin Othman, R.; Zhang, Z.

and Miyake, T. 2008. “Driving behavior classification

river sensing based on vehicle steering wheel

operations”.

SICE Annual Conference, 2008, 2714-2718.

Jensen, M. and Wagner, J. 2011. “Analysis of In-Vehicle

Driver Behaviour Data for Improved Safety”.

International Journal of Vehicle Safety. 197-212.

Li, W.; chang He, Q.; min Fan, X. and min Fei, Z.2012.

“Evaluation of driver fatigue on two channels of EEG

data”. Neuroscience Letters, 506, 235 – 239.

Lin, S. T.; Tan, Y. Y.; Chua, P. Y.; Tey, L. K. & Ang, C. H.

2012 “PERCLOS Threshold for Drowsiness Detection

during Real Driving” Journal of Vision, 12, 546.

Mohamad-Hoseyn Sigari, Mahmood Fathy and Soryani,

M. 2013. “A Driver Face Monitoring System for

Fatigue and Distraction Detection”. International

Journal of Vehicular Technology, 11.

Östlund, J; and al. 2004. “Delivrable 2- HMI and safety

related driver performance”. Human Machine

Interface and the Safety of Traffic in Europe (HASTE)

project.

Regan, M. A.; Hallett, C. and Gordon, C. P. 2011. “Driver

distraction and driver inattention: Definition,

relationship and taxonomy”. Accident Analysis &

Prevention, 43, 1771 -1781.

Santana Diaz, A.; Jammes, B.; Esteve, D. and Gonzalez

Mendoza, M. 2002. “Driver hypovigilance diagnosis

using wavelets and statistical analysis”. 2002.

Intelligent Transportation Systems. Proceedings. The

IEEE 5th International Conference on, 162-167

Trivedi, M. M.; G, T. and Mccall, J. 2007. “Looking-in

and looking-out of a vehicle: Computer-vision-based

enhanced vehicle safety”. IEEE Trans. Intell. Transp.

Syst, 8, 108-120.

Ueno, H.; Kaneda, M. and Tsukino, M. 1994.

“Development of drowsiness detection system”.

Vehicle Navigation and Information Systems

Conference, 1994. Proceedings. 1994, 15-20.

Wakayama, M.; Deguchi, D.; Doman, K.; Ide, I.; Murase,

H. and Tamatsu, Y.2012. ”Estimation of the human

performance for pedestrian detectability based on

visual search and motion features”.Pattern

Recognition (ICPR), 2012 21st International

Conference on, 1940-1943.

WHO, World Health Organization, 2013. “Global Status

Report on Road Safety Supporting a decade of action”.

EstimatingDriverUnawarenessofPedestrianbasedonVisualBehaviorsandDrivingBehaviors