A BCI Driving System to Understand Brain Signals Related to

Steering

Enrico Zero

, Simone Graffione

, Chiara Bersani

and Roberto Sacile

DIBRIS, Department of Informatics, Bioengineering, Robotics and Systems Engineering,

University of Genova, Genova, Italy

Keywords: EEG Signal, BCI, Arm Movements Recognition.

Abstract: In the last years, the manufactured vehicles were designed to focus on prevention of some risky situations

caused by a human driver. The aim of this paper is to illustrate the design and implementation of a BCI system

which can detect the arm movements by the EEG signal during a simulated driving session. The proposed

approach to realize a classifier able to recognize the arm movement by EEG feature analysis is based on the

consecutive application of a Time Delay Neural Network (TDNN) and a Pattern Recognition Neural Network

(PRNN). Preliminary tests are shown on three different participants between 24 and 45 years old.

1 INTRODUCTION

In the recent automotive market, the main challenging

activities are related to the realization of autonomous

vehicles (AV) (SAE, 2014). The SAE, On-Road

Automated Vehicle Standards Committee (SAE,

2014) classified the levels of automation according to

the different roles of the driver. Level 0 means the

vehicle is devoid of automation while the last level 5

refers to vehicle completely autonomous (Graffione

et al., 2020). In this latter case, any human

intervention or interaction is required during driving

task and the automated system monitors

autonomously the driving environment (Wang et al,

2021). The main important factor which contributed

to increase the attention of academic and industrial

scientists in the adoption of AV concerns mainly the

safety. Many studies demonstrated that the human

errors represent the first cause of road accidents

(Khattak et al., 2021, Bersani et al., 2012) and the

possibility to replace the driver with a partial or

complete advanced driver assistance system (ADAS)

may represent an opportunity for a safer road

transport system.

Above all for AV at the level 3 and 4 of

automation, the ADAS needs to be integrated by

https://orcid.org/0000-0002-9995-1724

https://orcid.org/0000-0003-0882-586X

https://orcid.org/0000-0002-5779-9605

https://orcid.org/0000-0003-4086-8747

intelligent systems which support the interaction

between the car driver and the AV (Graffione et al.,

2020, June). Special attention has to be dedicated to

the driver’s behaviour when he/she tackles critical

situation during the travel and his/her related reaction

to manage in real time the Human Machine Interface

(HMI). The importance of the human- machine

interaction in the context of AV inspired a field of

literature which aims to describe the driver’s mental

model in order to support the development of the AV

features.

The behavioural data coming from the user, in

fact, are essential to define a safer driving system

which has properly to intervene in real time during

critical traffic situations. Consequently, the ADAS

has to have the capability to evaluate and identify the

driver performance in order to recognize anomalous

behaviour.

In this context, the integration of Brain Computer

Interface (BCI) with HMI in the driving environment

have been introduced to determine the reaction of

drivers by brain activities when he/she makes driving

task. This approach supports the development of

systems able to identify motor intention of the driver

and, consequently, to produce commands in order to

support the user to prevent car accident (Gougeh et

Zero, E., Grafﬁone, S., Bersani, C. and Sacile, R.

A BCI Driving System to Understand Brain Signals Related to Steering.

DOI: 10.5220/0010576807450751

In Proceedings of the 18th International Conference on Informatics in Control, Automation and Robotics (ICINCO 2021), pages 745-751

ISBN: 978-989-758-522-7

745

al., 2021). The main promising technologies to

integrate the monitoring of the cognitive user’s state

with the external environment refers to the

electroencephalographic (EEG)-based BCI

applications (Gu et al., 2021). The driver, the vehicle,

the sensors on board, the HMI and the BCI represent

a complex system (Benza et al., 2012).

Different techniques may be used to analyse the

EEG features in terms of frequency, power or

bandwidth. The main objective is to define a proper

classifier able to correlate the EEG signals with the

related user’s cognitive state and, Artificial Neural

Network (ANN) appear one of the promising

methodologies to be applied. In (Chakole et al.,

2019), a review on EEG signals classification is

proposed while (Yger et al., 2017) presents a review

on BCI.

EEG consists of a special helmet equipped with

electrodes able to record the brain neural signals

which reproduce the user’s brain activities in the

different lobes (Zero et al, 2019). Large literature is

dedicated to the identification of human movement by

EEG signals related to human machine collaboration

with special reference to rehabilitation and robot

applications (Buerkle et al., 2021). Planelles et al.

(2014) compared different classifiers to identify the

user’s arm movements by his/her intention. They

concluded that support vector machine (SVM)

obtained a prediction accuracy of around 72% with

better performances in respect to k-nearest neighbour

(k-NN) and naive Bayes (NB) classifiers. Other

approaches confirmed the successful use of SVM to

recognize, by EGG signals, the human movements.

Classification accuracy in a range of 94%−97% is

verified to identify the motion intention to stand up,

sit and walk (Wang et al., 2018) while in Narayan Y.

(2021), the accuracy reached the 98% by SVM in

respect to linear discriminant analysis (LDA) and

multi-layer perceptron (MLP) classifiers to

discriminate left-hand and right-hand movements.

Other research groups proposed classifiers based on

the random forest technique (Kim et al., 2016), on

recurrent neural networks (RNN) (Idowu et al., 2021)

and on convolutional neural network (CNN) for hand

motor imagery tasks (Bressan et al., 2021) or linear

neural network, multilayer perceptron (MP), radial

basis function (RBF) network for the recognition of

imaginary movements of legs (Kurkin et al., 2018)

EEG based BCI is used in healthcare applications

but, recently, it has a significant research perspective

also in the context of safety transport. Interesting

works are published on the monitoring of driver’s

fatigue (Abbas et al., 2021, Borghini et al., 2014)

awareness (Kästle et al., 2021), distraction

(Taherisadr et al., 2019) or workload (Diaz-Piedra et

al., 2021). Limited literature is dedicated to the

identification of driver’s intent to compute

movements, as an example, to apply pressure on the

accelerator or on the braking pedals (Aydarkhanov et

al., 2021) or to detect braking intention in emergency

situations (Xing et al., 2019). Other approaches

focused on the recognition of driver’s actual actions

computed during the driving tasks. Zero et al. (2021a)

developed a time delay neural network (TDNN)

classification model to predict driver’s right and left

turns in a virtual driving environment. Zero et al.,

(2021b) adopted an ANN to recognize head yaw

rotations when the participant is subjected to visual

stimulus. The head positions directed toward a light

target were classified by three classes for the left and

the right positions and for the forward one. Bi at al.

(2016) integrated an extended queuing network

classifier in an EEG based BCI in order to convert the

desired steering wheel angle change into a desired

steering command for the vehicle.

This work has been developed in this latter

framework in order to realize a classifier able to

identify the driver’s arms motion on the right or on

the left while rotating the steering wheel of a car in a

simulated environment. The main objective is to

recognize, by the brain activities acquired by a EEG

helmet, the driver’s arm movements while he/she

carried out the turning task when the car have to

change the cruising line.

In this paper, a Time Delay Neural Network

(TDNN) classifier to recognize the direction of the

human arm movements is implemented and tested by

the adoption of EEG signals coming of six different

electrodes located in the central part of the brain. Due

to the complexity of this purpose, the output of the

TDNN is further elaborated by a pattern recognition

neural network (PRNN). The final results highlight a

strong correlation among the input

spectral EEG features and the targets associated to

the subject’s movements.

2 MODELS AND METHODS

2.1 Enobio Cap

A Enobio Cap (Enobio® EEG systems) with 8

channels is used in the EEG based BCI development.

In the proposed approach, the brain signals are

recorded by the following 6 electrodes according to

the International Standard System 10/20: C3, C1, CZ,

C5, C4, FZ. Figure. 1 shows the location of the used

electrodes (in orange are highlighted the selected

ICINCO 2021 - 18th International Conference on Informatics in Control, Automation and Robotics

746

electrodes). The electrodes are located in the central

area of the brain between the frontal and parietal

lobes. This part of the brain contains the primary and

secondary motor cortex areas which generate the

neural impulse for the execution of voluntary

movements (Bhattacharyya et al., 2011).

Figure 1: Electrode placement based on the simulation.

2.2 Simulation Environment

The simulation environment is based on two different

subsystems. Firstly, the BCI consists in a driving

interface available by a video screen. The second

system consists of a real car seat, a steering wheel and

the pedals which work as in a tradition car. This

second subsystem appears in the Figure 2.

The simulation environment was built using the

Add-On "Simulink 3D Simulation" of MATLAB®

software and it consists of a vehicle which travels on

one road with three lanes of 4 meters wide identified

by the following code: -1 (line on the left), 0 (central

line), 1 (line on the right). (See Figure 3).

The objective of the proposed test consists in

monitoring brain activities of a participant who have

to simulate a driving task. The proposed BCI provides

the user with a simulated environmental scenario,

displayed on a video, where a car rides on a straight

multilane road and it moves from one line to the

adjacent one. During the experiment, the BCI system

drives autonomously the vehicle across the different

carriage lanes. The participant, according to the car

path, has to move his arms in order to turn the real

steering wheel simulating the required changing line.

In detail, when the BCI system generates the

command and the car moves toward the new cruising

lane, this latter is highlighted in the environment

scenario, and the participant has to turn the steering

wheel according to the new requested vehicle

position.

Figure 2: Driving position.

Figure 3: Simulation environment (in the image, the line on

the right is highlighted to provide the user with the

information about the line to be covered and to simulate the

changing lane by turning the steering wheel).

During the simulation, the lane that the car has to

reach and keep is highlighted until a new turning is

requested. The vehicle may be moved with a lateral

speed of 0.8 𝑚/𝑠 and a longitudinal speed of 10 𝑚/𝑠.

During the training phase, a random sequence of

car movements for changing line is generated by the

system in order to induce the participant to turn the

steering wheel by his arms in order to follow the car

path. The simulation system works in real-time to

synchronize the car movement and the related EEG

signals recorded during the turning action.

A BCI Driving System to Understand Brain Signals Related to Steering

747

2.3 Test Driver

The simulations have been performed by three

different men between 24 and 45 years old with

driving license. The simulation was 5 minutes long.

During the test the participants was sitting in a driver

seat in front of a LCD screen where the simulation

environment was projected.

When the simulation started, the participant

followed the car on the screen. When a new line to be

covered was indicated and the car started the

movement, the user had to turn the steering wheel,

held by his hands, by rotating his arms to make the

turning motion according to the car behaviour on the

screen, as shown in Figure 4.

Figure 4: Driving simulation.

2.4 Elaboration Data

The EEG signals recorded by the NIC software,

property of Enobio, have been saved and elaborated

by Matlab R2020b. After the synchronization

between the signals regarding the brain activity and

the car movements, a bandstop/notch filter between

49 Hz and 51 Hz is applied to EEG signals in order to

delete the noises produced by the electric components

of the devices. Besides, a high pass filtering to 1 Hz

is performed to remove the common components.

In the proposed work, two different classifiers

have been applied to categorize the EEG signals.

Firstly, the signals have been processed by a TDNN

according to the input-output function (1)

𝑦



𝑡



=𝑓



𝑥



𝑡1



,𝑥



𝑡2



,…𝑥𝑡10



(1)

where 𝑦



𝑡



is the codified arm movements

prediction (-1, 0, 1) and 𝑥



𝑡



represent the input EEG

signal coming from the six selected electrodes.

According to the non-linear function 𝑓in (1), the

TDNN produces a classification which is the input of

the PRNN. The proposed architecture of the system

appears in the figure 5.

The TDNN is specific class of feedforward neural

network which works on sequential data and it is able

to realize the recognition of the EEG patterns or

features. The output of the TDNN is a real number

and the PRNN related to the pattern recognition

classifies the values into one of the three classes.

Figure 5: Architecture.

The used TDNN consists of 10 time delays and 10

hidden fields. The proposed PRNN, which consists of

three neurons, solves a data classification problem

using a two-layer feed-forward network. It provides a

statistical analysis of the TDNN output in order to

assign the different patterns to the different categories

represented by the targets.

In this way, the EEG signals may be assigned to

the correct pattern and they are classified according

to the three classes of arm movements such as -1

(left), 0 (centre), 1 (right).

The performance indexes used to evaluate the

accuracy of the prediction generated by the TDNN

are the correlation and the MSE values. Regarding the

PRNN, cross-entropy will be analysed. The cross-

entropy (CE) aims to minimize the loss function used

in the NN. Besides, it measures the distance between

the output probabilities and the related true values

with the objective to make the model output closest

to the expected values. In the training phase, the

model weights are continuously adjusted in order to

minimize the cross-entropy loss. Thus, a reliable

model should have a CE loss near to 0.

3 RESULTS

Table 1 shows the prediction performance indexes

related to the TDNN for each participant. In this case,

the 50% of the EEG signals are used the training phase

and the remaining 50% for the test.

ICINCO 2021 - 18th International Conference on Informatics in Control, Automation and Robotics

748

The correlation indexes, evaluated according to

Cohen classification (Cohen, 1988), demonstrate a

strong linear correlation in two of a three participants.

Table 1: Prediction performance for the TDNN.

PARTECIPANT ID

R MSE

1 0.75 0.237

2 0.65 0.310

3 0.30 0.312

Also MSE values demonstrate good

performances in the prediction considering as target

the actual car changing lines and the predicted value

of the related arm movements to turn the steering

wheel realized in the last 2,5 minutes of the

simulations.

In the Fig. 6, a comparison between the target

values and the predicted ones appears. In the

proposed image, the line associated to the prediction

presents positive and negative peaks in accordance to

the targets associated to the car changing line. Value

1, for the real pattern, represented a request to change

line on the right while value -1 a turning on the left.

This means that for each line shift of the car along the

route, the TDNN identifies, by the human brain

activities, a participant’s cognitive reaction to realize

the correct driving task to turn the steering wheel by

his arms.

Figure 6: TDNN results.

The results related to the PRNN application are

displayed in the Table 2. In second column, the

performances of the classifier have been evaluated by

the accuracy percentage of the correct classified

observations computed as follows:

%𝐴𝑐𝑐𝑢𝑟𝑎𝑛𝑐𝑦









*100

where

𝑁



=Number of samples assigned to the correct

class;

𝑁



= Total number of samples used in the

testing phase.

The accuracy of the model appears significantly

relevant in the three cases with a mean value of about

79% of correct predictions.

Table 2: Prediction performance by PRNN.

PARTECIPANT ID

Accuracy

1 81.25 2.24

2 80.40 1.80

3 74.10 1.32

In the third column of the Table II, the

performance index associate to the cross-entropy

(CE) appears. The results are inferior to the value 2 in

two of the three cases underlying good performances

for the proposed approach.

In addition, the confusion matrix associated to the

first participant is provided in Fig. 7. In the diagonal

of the matrix, the correct prediction of the class (in

bolt) appears with the related percentage of

correctness in respect to the overall analysed features

that, in the example, are 224. Out of the diagonal, the

absolute values and percentage of the wrong

predictions related to the different classes are shown.

Figure 7: Confusion matrix for the Participant 1.

As an example, in the first row of the matrix, the

value 19 (8,5%) means that for 19 times during the

test phase (which considers 2,5 minutes of the

A BCI Driving System to Understand Brain Signals Related to Steering

749

experiment with a total of 224 requested driving

tasks), the classifier associated correctly the driver’s

arm movement on a left turning (output value= -1)

when, in the real scenario, an actual left movement

has been realized (target value= -1).

It is possible to note that, never, a completely

wrong prediction about the turning left/right has been

detected, in fact, in the matrix, for the couples (-1,1)

and (1, -1), the values are 0 (0%).

4 CONCLUSIONS

The focus of the work is related to the implementation

of BCI based classifier able to identify the human arm

movements by the brain activities of a driver who has

to turn a real steering wheel following a car which

changes line on a straight multilane road visualized in

a simulated scenario.

The proposed BCI acquires the brain signals by a

EEG cap worn by the participants who have to carry

out the requested driving tasks. The signals are pre-

processed in order to limit the artefacts and then two

different NNs are applied to generate the human arm

movements classification.

The analysis of the output coming by the TDNN

and the PRNN demonstrated a good correlation

among the input brain signals and the output related

to the driver’s movements codified by three different

classes associated to the changing line on the right, on

left or to continue the path on the central line.

Further efforts will be dedicated to the pre-

processing elaboration data in order to filter the

component of the EEG signals not correlated to the

human brain activities. Besides, a large set of

participants have to be involved to validate the

proposed architecture for the classifier model.

ACKNOWLEDGEMENTS

This work has been partially sponsored by Eni S.p.A.,

under a research agreement with University of

Genova, Italy.

REFERENCES

Abbas, Q., & Alsheddy, A. (2021). Driver fatigue detection

systems using multi-sensors, smartphone, and cloud-

based computing platforms: a comparative analysis.

Sensors, 21(1), 56.

Aydarkhanov, R., Uscumlic, M., Chavarriaga, R.,

Gheorghe, L., & Millan, J. D. R. (2021). Closed-loop

EEG study on visual recognition during driving.

Journal of neural engineering.

Benza, M., Bersani, C., D'Incà, M., Roncoli, C., Sacile, R.,

Trotta, A., ... & Ridolfi, R. (2012, July). Intelligent

transport systems (its) applications on dangerous good

transport on road in italy. In 2012 7th International

Conference on System of Systems Engineering (SoSE)

(pp. 223-228).

Bersani, C., & Roncoli, C. (2012, July). Real-time risk

definition in the transport of dangerous goods by road.

In 2012 7th International Conference on System of

Systems Engineering (SoSE) (pp. 131-136).

Bhattacharyya, S., Khasnobish, A., Konar, A., Tibarewala,

D. N., & Nagar, A. K. (2011, April). Performance

analysis of left/right hand movement classification

from EEG signal by intelligent algorithms. In 2011

IEEE Symposium on Computational Intelligence,

Cognitive Algorithms, Mind, and Brain (CCMB) (pp. 1-

8). IEEE.

Bi, L., Lu, Y., Fan, X., Lian, J., & Liu, Y. (2016). Queuing

network modeling of driver EEG signals-based steering

control. IEEE Transactions on Neural Systems and

Rehabilitation Engineering, 25(8), 1117-1124.

Borghini, G., Astolfi, L., Vecchiato, G., Mattia, D., and

Babiloni, F. (2014). Measuring neurophysiological

signals in aircraft pilots and car drivers for the

assessment of mental workload, fatigue and

drowsiness. Neurosci. Biobehav. Rev. 44, 58–75. doi:

10.1016/j.neubiorev.2012.10.003

Bressan, G., Cisotto, G., Müller-Putz, G. R., &

Wriessnegger, S. C. (2021). Deep learning-based

classification of fine hand movements from low

frequency EEG. Future Internet, 13(5), 103.

Buerkle, A., Eaton, W., Lohse, N., Bamber, T., & Ferreira,

P. (2021). EEG based arm movement intention

recognition towards enhanced safety in symbiotic

Human-Robot Collaboration. Robotics and Computer-

Integrated Manufacturing, 70, 102137.

Chakole, A. R., Barekar, P. V., Ambulkar, R. V., &

Kamble, S. D. (2019). Review of EEG signal

classification. In Information and Communication

Technology for Intelligent Systems (pp. 105-114).

Springer, Singapore.

Cohen, L. H. (1988). Life events and psychological

functioning: Theoretical and methodological issues

(Vol. 90). Sage Publications, Inc.

Diaz-Piedra, C., Rieiro, H., & Di Stasi, L. L. (2021).

Monitoring army drivers’ workload during off-road

missions: An experimental controlled field study.

Safety science, 134, 105092.

Enobio® EEG systems. [online] Available at

https://www.neuroelectrics.com/solutions/enobio. Last

access March 2021.

Gougeh, R. A., Rezaii, T. Y., & Farzamnia, A. (2021). An

Automatic Driver Assistant Based on Intention

Detecting Using EEG Signal. In Proceedings of the

11th National Technical Seminar on Unmanned System

Technology 2019 (pp. 617-627). Springer, Singapore.

Gu, X., Cao, Z., Jolfaei, A., Xu, P., Wu, D., Jung, T. P., &

Lin, C. T. (2021). Eeg-based brain-computer interfaces

ICINCO 2021 - 18th International Conference on Informatics in Control, Automation and Robotics

750

(bcis): A survey of recent studies on signal sensing

technologies and computational intelligence

approaches and their applications. IEEE/ACM

Transactions on Computational Biology and

Bioinformatics.

Graffione, S., Bersani, B., Sacile, R., & Zero, E. (2020).

Model Predictive Control for Cooperative Insertion or

Exit of a Vehicle in a Platoon. In Proceedings of the

17th International Conference on Informatics in

Control, Automation and Robotics (ICINCO’20) (pp.

352-359).

Graffione, S., Bersani, C., Sacile, R., & Zero, E. (2020,

June). Model predictive control of a vehicle platoon. In

2020 IEEE 15th International Conference of System of

Systems Engineering (SoSE) (pp. 513-518).

Idowu, O. P., Ilesanmi, A. E., Li, X., Samuel, O. W., Fang,

P., & Li, G. (2021). An Integrated Deep Learning

Model for Motor Intention Recognition of Multi-Class

EEG Signals in Upper Limb Amputees. Computer

Methods and Programs in Biomedicine, 106121.

Kästle, J. L., Anvari, B., Krol, J., & Wurdemann, H. A.

(2021). Correlation between Situational Awareness and

EEG signals. Neurocomputing, 432, 70-79.

Khattak, A. J., Ahmad, N., Wali, B., & Dumbaugh, E.

(2021). A taxonomy of driving errors and violations:

Evidence from the naturalistic driving study. Accident

Analysis & Prevention, 151, 105873.

Kim Y., Ryu J., Kim K.K., Took C.C., Mandic D.P., C. Park

Motor imagery classification using Mu and Beta

rhythms of EEG with strong uncorrelating transform

based complex common spatial patterns. Comput.

Intell. Neurosci., 2016 (2016), Article 1489692,

10.1155/2016/1489692.

Kurkin, S. A., Pitsik, E. N., Musatov, V. Y., Runnova, A.

E., & Hramov, A. E. (2018). Artificial Neural Networks

as a Tool for Recognition of Movements by

Electroencephalograms. In ICINCO (1) (pp. 176-181).

Narayan, Y. (2021). Motor-Imagery EEG Signals

Classificationusing SVM, MLP and LDA Classifiers.

Turkish Journal of Computer and Mathematics

Education (TURCOMAT), 12(2), 3339-3344.

Planelles, D., Hortal, E., Costa, Á., Úbeda, A., Iáez, E., &

Azorín, J. M. (2014). Evaluating classifiers to detect

arm movement intention from EEG signals. Sensors,

14(10), 18172-18186.

SAE On-Road Automated Vehicle Standards Committee.

(2014). Taxonomy and definitions for terms related to

on-road motor vehicle automated driving systems. SAE

Standard J3016 201401. Warrendale, PA: SAE

International.

Taherisadr, M., & Dehzangi, O. (2019). Eeg-based driver

distraction detection via game-theoretic-based channel

selection. In Advances in Body Area Networks I (pp. 93-

105). Springer, Cham.

Wang J., Huang H., Li, K. & J. Li, (2021). Towards the

Unified Principles for Level 5 Autonomous Vehicles.

Engineering, 2021.

Wang, C., Wu, X., Wang, Z., & Ma, Y. (2018).

Implementation of a brain-computer interface on a

lower-limb exoskeleton. IEEE access, 6, 38524-38534.

Xing, Y., Lv, C., Wang, H., Cao, D., Velenis, E., & Wang,

F. Y. (2019). Driver activity recognition for intelligent

vehicles: A deep learning approach. IEEE transactions

on Vehicular Technology, 68(6), 5379-5390.

Yger F., Berar M. and Lotte F., "Riemannian Approaches

in Brain-Computer Interfaces: A Review," in IEEE

Transactions on Neural Systems and Rehabilitation

Engineering, vol. 25, no. 10, pp. 1753-1762, Oct. 2017,

doi: 10.1109/TNSRE.2016.2627016.

Zero, E., Bersani, C., Zero, L., & Sacile, R. (2019).

Towards real-time monitoring of fear in driving

sessions. IFAC-PapersOnLine, 52(19), 299-304.

Zero, E., Bersani, C., & Sacile, R. (2021a). EEG Based BCI

System for Driver’s Arm Movements Identification.

Automation, Robotics & Communications for Industry

4.0, 77.

Zero, E.; Bersani, C.; Sacile, R. (2021b) Identification

of Brain Electrical Activity Related to Head

Yaw Rotations. Sensors 2021, 21, 3345.

https://doi.org/10.3390/s21103345

A BCI Driving System to Understand Brain Signals Related to Steering

751