Traffic Sign Recognition System based on Belief Functions Theory

Nesrine Triki

, Mohamed Ksantini

and Mohamed Karray

National School of Engineers of Sfax, University of Sfax, Sfax, Tunisia

ESME Sudria, The Embedded and Electronic Systems Lab, Ivry Sur Seine, France

Keywords: Advanced Driver Assistance Systems (ADAS), Autonomous Vehicles, Traffic Sign Recognition, Belief

Functions, Artificial Intelligence, Machine Learning, Image Processing.

Abstract: Advanced Driver Assistance Systems (ADAS) have a strong interest in road safety. This type of assistance

can be very useful for collision warning systems, blind spot detection and track maintenance assistance.

Traffic Sign Recognition system is one of the most important ADAS technologies based on artificial

intelligence methodologies where we obtain efficient solutions that can alert and assist the driver and, in

specific cases, accelerate, slow down or stop the vehicle. In this work, we will improve the effectiveness and

the efficiency of machine learning classifiers on traffic signs recognition process in order to satisfy ADAS

reliability and safety standards. Hence, we will use MLP, SVM, Random Forest (RF) and KNN classifiers on

our traffic sign dataset first, each classifier apart then, by fusing them using the Dempster-Shafer (DS) theory

of belief functions. Experimental results confirm that by combining machine learning classifiers we obtain a

significant improvement of accuracy rate compared to using classifiers independently.

1 INTRODUCTION

Recent technological advancements are expected to

steer growth in favour of the global autonomous car

industry through 2025. With the emergence of

advanced technology, automakers are expected to

invest heavily in autonomous and electronic vehicle

technology. For instance, “Waymo” began as the

Google Self-Driving Car (Google car) Project in 2009

and has been testing its vehicles since early 2017 until

now (Waymo Safety Report, 2017).

The continuing evolution of automobile

technology aims to deliver even greater safety

benefits and automated driving systems that can one

day handle the whole task of driving when we don’t

want to or can’t do it ourselves. In this context, Traffic

Signs Recognition (TSR) System is considered one of

the most important Advanced Driver Assistance

Systems (ADAS) technologies. It can assist drivers or

be part of automatic driving systems in real time in

order to facilitate the driving process and optimize the

level of safety and comfort on the road.

In fact, in the driving environment, traffic sign

types and patterns are incoherent in various countries.

https://orcid.org/0000-0002-2770-2526

https://orcid.org/0000-0002-9928-8643

https://orcid.org/0000-0001-7293-8696

Hence, the TSR system uses the combined feature of

shape and colour to identify and recognize traffic

signs into many categories such as warning,

regulatory and informative signs. Table 1 shows the

different types of signs used in European roads.

Table 1: European traffic sign categories definition (main

categories and shapes).

Danger/Warning Regulatory Informative

TSR system uses various methods that first detect

and extract the candidates’ regions of the traffic sign

(ROIs) and then classify them according to

predefined classes.

Several methodologies have already been applied

to image recognition and have given good results.

However, they still suffer from such problems like

losing some details of the image when extraction

image features and the ineffectiveness of the used

classifier. In fact, the loss of information is due to

Triki, N., Ksantini, M. and Karray, M.

Trafﬁc Sign Recognition System based on Belief Functions Theory.

DOI: 10.5220/0010239807750780

In Proceedings of the 13th International Conference on Agents and Artiﬁcial Intelligence (ICAART 2021) - Volume 2, pages 775-780

ISBN: 978-989-758-484-8

775

many forms of external noise like fading and blurring

effect, affected visibility, multiple appearances of the

sign, chaotic background, viewing angle problem,

damaged and partially obscured sign, etc. Hence, we

find ourselves in a situation of indecision where even

the use of machine learning algorithms for the

classification of traffic sign images does not solve the

problem of uncertainty.

Therefore, in order to overcome this limitation,

we propose, in this paper, the use of Dempster-Shafer

(DS) theory of belief functions (Dempster, 1967),

(Shafer, 1976) to make decisions on detected signs

with uncertainty (Yager and Liu., 2008). In order to

classify ROI images, the DS rules of classifiers fusion

will be used by combining the outcomes provided

from MLP, RF, KNN and SVM classifiers.

This paper will be structured as follows: section 2

will present the machine learning algorithms and the

DS theory applied to traffic sign images. Section 3

will describe our proposed contribution to classify

detected traffic signs. Section 4 will show

experimental results and discussion. In the

conclusion, we will propose some perspectives to

extend this work.

2 MACHINE LEARNING AND

DS-THEORY

A lot of works were proposed to deal with image

classification through different machine learning

methods and belief functions, especially the

Dempster-Shafer (DS) theory which is based on rule

combination and fusion of classifiers.

2.1 Machine Learning

A machine learning classifier is a system able to

predict the class of a phenomenon being observed.

The use of a classifier depends on the application

and the nature of available data set. In (Ksantini, Ben

Hassena and Delmotte, 2017) authors present a

comparison between ML classifiers according to 5

criteria: Speed of classification, accuracy, tolerance

to noise and Robustness.

There are a variety of applications areas in which

ML classifiers can be applied like road safety. In this

field of application, several ML classifiers provide

good accuracy rates like:

 Multiple Layer Perceptions classifier (MLP) is a

single-layer neural network organized in a

cascade and subdivided in an input layer, one or

more hidden layers and an output layer.

(Genevieve, Taif and Wasfy,2019)

 K-nearest neighbour classifier (KNN) is based on

a distance function that calculates similarity

between the object to classify and its neighbours.

(Karthiga, Mansoor and Kowsalya, 2016)

 Support Vector Machine classifier (SVM) is

based on the statistical learning theory. Thus, the

goal of this method is a binary classification of

data. (Anusha and Renuka, 2019)

 Random Forest classifier (RF) is an ensemble of

classification trees, where each tree contributes

with a single vote for the assignment of the most

frequent class to the input data. (Ellahyani, El

Ansari and El Jaafari, 2016)

In (Wahyono and Kang-Hyun, 2014), authors

employed SVM, RF, KNN and MLP for three types

of traffic sign recognition (warning, prohibition and

mandatory) from the German Traffic Sign Dataset.

Achieved accuracy was 78.7%, 76.3%, 76.3% and

70% for KNN, SVM, RF and MLP classifiers,

respectively.

Authors in (Gomes, Rebouças and Neto, 2016)

presented obtained accuracy rates of several

classifiers for the recognition of the segmented speed

limit digits for embedded applications. Obtained

results were 87.12%, 97.04%, 98.51% for MLP, SVM

and KNN classifiers respectively.

We notice that machine learning algorithms were

used to classify different types of traffic sign images

with proportional accuracy rates that can be improved

by using the strengths of one method to complete the

weakness of another algorithm. In the next part we

introduce belief functions theory which has been

applied to pattern recognition and specially to

supervised classification.

2.2 DS Theory

In the previous part, we have presented the

classification algorithms which can deal only with

certain and complete information. So, in this part we

will treat the case of uncertain data by using belief

functions theory.

The Dempster-Shafer (DS) theory of belief

functions was introduced by Arthur P. Dempster in

the context of statistical inference in 1968, and was

later developed by Glenn Shafer in 1976 as a theory

of evidence. This theory represents the formalism for

making decisions with uncertainty. It has been

applied to supervised and unsupervised classification

(Thierry, 2019).

In (Xu, Krzyzak and Suen, 1992) and (Liu, Pan,

Dezert, Han, and He, 2018), the outputs of classifiers

ICAART 2021 - 13th International Conference on Agents and Artiﬁcial Intelligence

776

have been expressed by the formalism of belief

functions and have been combined with the

Dempster’s rule in the case of classifier fusion.

In (Xu, Davoine, Zha and Denœux, 2016)

(Minary, Pichon, Mercier, Lefevre and Droit, 2017),

the used approach was the conversion of the decisions

obtained from classifiers such as the conversion of the

SVM into belief functions.

The basics of DS theory are:

 Mass function m: represents an element of

evidence X with value in Ω and m (A) quantifies

the belief allocated to the proposition. It is defined

by:







m: 2 0 ,1 with m A 1

A





Ω

(1)

 Correction of the information: the mass function

m and the belief degree in the reliability of the

source µ. The new mass function of the weakness

operation is:









m A µ* m A ; A Ω

(2)

 Information fusion: the mass function m1

obtained from the source S1 and the mass function

m2 obtained from the source S2. The new mass

function after the use of Dempster’ rule is defined

by:

  

m(m1 m2 A)C m1 * 2B

AB C A B





(3)

 Decision making: we need a pignistic

transformation which represents the probability

distribution obtained from the fusion result. This

transformation is defined by:



{A , ω A}

)

((

)

p ω

1m A









Ω

(4)

The decision will be made by choosing the

element x with the greatest probability from pignistic

transformation:













Rp x argmax Betp ω x





Ω

(5)

3 PROPOSED METHODOLOGY

Our methodology is based on 3 parts:

 Data processing

 Traffic sign classification using machine learning

classifiers (MLP, SVM, Random Forest and

KNN)

 Traffic sign classification using DS theory of

belief functions for classifiers fusion.

3.1 Data Processing

Given the diversity of road sign pictograms for each

country and due to the lack of a French traffic sign

dataset Benchmark, we are led to build a dataset

mixing road signs images from the German Traffic

Sign Dataset and images generated from image

processing codes.

We have built a dataset containing 26560 images

divided into 15 classes (8 speed limit classes and 7

Mandatory traffic signs classes) shown in figure 1. In

fact, the number of samples per class varies from one

class to another. The top class (Speed limit

30km/h) has over 3500 examples while the least

represented class (Go straight or left) has fewer than

500 examples. This unbalanced dataset depends from

the training process.

Our dataset will be gradually incremented in order

to reach all the road signs pictograms.

Figure 1: Distribution of traffic sign images.

According to traffic sign image classification; we

used histogram of oriented gradient (HOG) feature

descriptor in order to extract features from the image.

The main reasons to use it are that it is accurate and

fast and we can easily run the program on a CPU. In

fact, gradients (x and y derivatives) of an image are

useful because the magnitude of gradients is large

around edges and it is known that edges and corners

pack in a lot more information about object shape

than flat regions so the gradient intensities of an

image can reveal some useful local information that

can lead to recognition of the image (Reinaldo,

Manurung, Simbolon and Christnatalis, 2019).

Trafﬁc Sign Recognition System based on Belief Functions Theory

777

After that, machine learning algorithms and DS

theory are applied to determine the best accuracy rate

for traffic sign classification.

3.2 Traffic Signs Classification using

Machine Learning Algorithms

In order to train and test MLP, SVM, RF and KNN

classifiers, we first calculate HOG descriptor for

every image in the dataset. Then, we split data into a

training set (90%) and a testing set (10%). Finally, we

save the trained model obtained in order to validate it

on other traffic sign images detected from a real time

camera.

Experimental results shown in Figure 2, 3, 4 and

5 are presented in the form of confusion matrix which

is a table with 4 different combinations of predicted

and actual values (TP, TN, FP and FN):

 True Positives (TP): The number of positive

instances that were classified as positive.

 True Negatives (TN): The number of negative

instances that were classified as negative.

 False Positives (FP): The number of negative

instances that were classified as positive.

 False Negatives (FN): The number of positive

instances that were classified as negative.

These combinations are extremely useful for

measuring Precision, Recall and Accuracy rate:

 Precision, often referred to as positive predictive

value, is the ratio of correctly classified positive

instances to the total number of instances

classified as positive:

True positive

Precision

True Positive False Positive





(6)

 Recall, also called sensitivity or true positive rate,

is the ratio of correctly classified positive

instances to the total number of positive instances:

True positive

Recall

True Positive False Negative





(7)

 F1 combines precision and recall as single value:

Precision * Recall

F1 2*

Precision Recall





(8)

We note that all results are obtained with a PC

having the hardware configuration: Intel® Core (TM)

i5-7200 CPU, 64 bits; RAM: 8GB.

We notice from the previous confusion matrix that

the accuracy rate is important: MLP 86%, SVM:

83%, Random Forest: 83% and KNN: 81%. Despite

these results, the traffic sign recognition system must

Figure 2: MLP Confusion matrix.

Figure 3: SVM Confusion matrix.

Figure 4: RF Confusion matrix.

Figure 5: KNN Confusion matrix.

ICAART 2021 - 13th International Conference on Agents and Artiﬁcial Intelligence

778

have the minimum of possible errors in order to

ensure road safety requirements.

Therefore, we choose to use DS theory in order to

improve further the accuracy rate of TSR system.

3.3 DS Theory

Given the mass function for every classifier of the DS

theory (m1, m2, m3 and m4) for respectively MLP,

SVM, RF and KNN classifiers, we have got 11 types

of combination (data fusion) between classifiers

based on the mass’s combination DS rule:

 m1 ⊕ m2

 m1 ⊕ m3

 m1 ⊕ m4

 m2 ⊕ m3

 m2 ⊕ m4

 m3 ⊕ m4

 m1 ⊕ m2 ⊕m3

 m1 ⊕ m2 ⊕m4

 m1 ⊕ m3 ⊕m4

 m2 ⊕ m3 ⊕m4

 m1⊕ m2 ⊕ m3 ⊕m4

The pignistic transformation of the obtained

masses helps us to make a decision about the obtained

algorithms after fusion.

4 EXPERIMENTAL RESULTS

AND DISCUSSION

Before using information fusion, each classifier has a

degree of weakness. Table 2 represents the error rate

of the classifiers MLP, SVM, RF and KNN on

validation data set.

Table 2: Degrees of weakness of MLP, SVM, RF and KNN.

Classifier MLP SVM RF KNN

Weakness Degree 13.97 17.09 17.17 18.64

In our experiments, we have used the Dempster-

Shafer theory in fusion of two, three and four machine

learning classifier outputs in order to decrease the

weakness degrees. Hence, by using (2), the new mass

functions were then combined by Dempster-Shafer

rule (3) associating 2, 3 and 4 classifiers.

The decision was made by the pignistic risk (4)

and (5) that shows the performance of belief functions

in terms of error rate in test set. Obtained results are

summarized in Table 3, 4 and 5.

Table 3: Degrees of Accuracy and Weakness after

combining two classifiers.

Combined

Classifiers

Accuracy

Rate

Degrees of weakness

of fusion

MLP and KNN 97.37 % 2.63 %

MLP and RF 97.56 % 2.44 %

MLP and SVM 97.53 % 2.47 %

KNN and SVM 97.41 % 2.59 %

KNN and RF 97.44 % 2.56 %

SVM and RF 97.54 % 2.46 %

Table 4: Degrees of Accuracy and Weakness after

combining three classifiers.

Combined

Classifiers

Accuracy

Rate

Degrees of weakness

of fusion

MLP and SVM and

KNN

99.32 % 0.68 %

MLP and SVM and

99.34 % 0.66 %

MLP and KNN and

99.33% 0.67 %

SVM and KNN and

99.33% 0.67%

Table 5: Degrees of Accuracy and weakness after

combining four classifiers.

Combined

Classifiers

Accuracy

Rate

Degrees of weakness

of fusion

MLP and SVM and

KNN and RF

99.85 % 0.15

As a conclusion, the results obtained from the

different confusion matrix of different classifiers

have shown that the accuracy of these algorithms is

excellent for predicting obligation traffic signs (keep

right, ahead only, turn right, turn left, keep left, go

straight or left) and good for some speed limit signs

(20 km/h, 70 km/h and 80km/h) but some problems

have appeared for identifying the other speed limit

traffic signs correctly.

In addition to that, we have shown that the

information fusion has the lowest error rate in

comparing with other classifiers. So, the accuracy

obtained using Dempster-Shafer theory in traffic sign

classification is better than that obtained using

machine learning classifiers independently.

5 CONCLUSION AND

PERSPECTIVES

Driving assistance systems can help drivers and

automatic driving systems to avoid the occurrence of

a dangerous situation that could lead to an accident,

Trafﬁc Sign Recognition System based on Belief Functions Theory

779

free the driver from a number of tasks that could

reduce their vigilance and assist him in his perception

of the environment. Therefore, safety and reliability

validation of Advanced Driver Assistance Systems

(ADAS) is strongly recommended.

In this paper, we have proposed a methodology

based on Machine Learning algorithms and belief

functions theory to improve the performance of TSR

systems. We carried out a combinatorial study of

several classifier outputs in order to find the best

combination leading to this improvement. For this,

we have classified data into 15 sets based on

pictograms. Then, firstly, we have used machine

learning algorithms (MLP, SVM, RF and KNN) to

classify detected signs. Secondly, we have applied DS

theory by combining 2, 3 and 4 of the previous

classifiers. This methodology has given us better

results than using the different classifiers each one

apart.

As perspectives, we will extend our traffic sign

dataset by other classes in order to obtain a full French

traffic sign dataset then we would apply Dempster-

Shafer theory on deep learning algorithms and

compare obtained results with this work.

REFERENCES

Anusha, A., Renuka Devi, S.M., 2019. Automatic Speed-

Limit Sign Detection and Recognition for Advanced

Driver Assistance Systems. International Journal of

Innovative Technology and Exploring Engineering

(IJITEE) ISSN: 2278-3075, Volume-8, Issue- 9S2.

Dempster, A. P., 1967. Upper and lower probabilities

induced by a multivalued mapping. Annals of

Mathematical Statistics, vol. 38, pp. 325–339.

Ellahyani, A., El Ansari, M., El Jaafari, I., 2016. Trafﬁc

sign detection and recognition based on random forests.

Appl Soft Comput 46:805–815.

Genevieve, S., Taif, A., Wasfy, B.M.,2019. Traffic Sign

Recognition Based on Multilayer Perceptron Using

DWT and DCT. IEEE 62

International Midwest

Symposium on Circuits and Systems (MWSCAS)

Gomes, SL., Rebouças, ES, Neto EC., 2016. Embedded

real-time speed limit sign recognition using image

processing and machine learning techniques. The

Natural Computing Applications Forum.

Karthiga, PL., Mansoor, R., Kowsalya, J., 2016. Traffic-

sign recognition for an intelligent vehicle/driver

assistant system using HOG. Computer Science &

Engineering, An International Journal (CSEIJ), Vol.6,

No.1.

Ksantini, M., Ben Hassena, A., Delmote, F., 2017.

Comparaison and fusion of classifiers applied to a

medical diagnosis. 14th international Multi-

Conference on Systems, Signals & Devices.

Liu, Z., Pan, Q., Dezert, J., Han, J. W., He, Y., 2018.

Classifier fusion with contextual reliability evaluation,

IEEE Transactions on Cybernetics, Volume:

48, Issue:5.

Minary, P., Pichon, F., Mercier, D., Lefevre, E., Droit, B.,

2017. Face pixel detection using evidential calibration

and fusion, International Journal of Approximate

Reasoning, vol. 91, pp. 202–215.

Reinaldo, Natanael, M., Juara, I., S., Christnatalis, 2019.

Traffic sign detection using histogram of oriented

gradients and max margin object detection, Journal of

Physics : Conf. Series 1230 (2019) 012098.

Shafer, G., 1976. A mathematical theory of evidence.

Princeton University Press.

Thierry, D., 2019. Logistic Regression, Neural Networks

and Dempster-Shafer Theory: A New Perspective.

Knowledge-Based Systems, Elsevier, 176, pp.54-67.

Wahyono, W., Kang-Hyun, J., 2014. A Comparative Study

of Classification Methods for Traffic Signs

Recognition, IEEE International Conference on

Industrial Technology (ICIT), Busan, Korea

Xu, L., Krzyzak, A., Suen, C., 1992. Methods of combining

multiple classifiers and their applications to

handwriting recognition, IEEE Transactions on

Systems, Man and Cybernetics, vol. 22, no. 3, pp. 418–

435.

Xu, P., Davoine, F., Zha, H., Denœux, T., 2016. Evidential

calibration of binary SVM classifiers, International

Journal of Approximate Reasoning, vol. 72, pp. 55–70

Yager, RR., Liu, L., 2008. Classic Works of the Dempster

Shafer Theory of Belief Functions. Heidelberg,

Springer. 36.

Waymo Safety Report, 2017. On the road to fully self-

driving.

ICAART 2021 - 13th International Conference on Agents and Artiﬁcial Intelligence

780