Intrusion Detection in Wi-Fi Networks

by Modular and Optimized Ensemble of Classiﬁers

Giuseppe Granato

, Alessio Martino

, Luca Baldini

and Antonello Rizzi

Department of Information Engineering, Electronics and Telecommunications, University of Rome “La Sapienza”,

via Eudossiana 18, 00184 Rome, Italy

Keywords:

Information Granulation, Data Clustering, Supervised Learning, Genetic Algorithms, Malicious Trafﬁc

Detection, Network Intrusion Detection Systems.

Abstract:

With the breakthrough of pervasive advanced networking infrastructures and paradigms such as 5G and IoT,

cybersecurity became an active and crucial ﬁeld in the last years. Furthermore, machine learning techniques

are gaining more and more attention as prospective tools for mining of (possibly malicious) packet traces and

automatic synthesis of network intrusion detection systems. In this work, we propose a modular ensemble

of classiﬁers for spotting malicious attacks on Wi-Fi networks. Each classiﬁer in the ensemble is tailored

to characterize a given attack class and is individually optimized by means of a genetic algorithm wrapper

with the dual goal of hyper-parameters tuning and retaining only relevant features for a speciﬁc attack class.

Our approach also considers a novel false alarm management procedure thanks to a proper reliability measure

formulation. The proposed system has been tested on the well-known AWID dataset, showing performances

comparable with other state of the art works both in terms of accuracy and knowledge discovery capabilities.

Our system is also characterized by a modular design of the classiﬁcation model, allowing to include new

possible attack classes in an efﬁcient way.

1 INTRODUCTION

Recent developments in wireless networking technol-

ogy have been key elements in the evolution of fu-

ture smart environments, and in the last few years

technology has deeply evolved to fulﬁl needs in dif-

ferent areas. Starting from enterprise wireless net-

works, considering also Internet of Thing (IoT) net-

works, the introduction of new smart services implied

the birth of new attack scenarios. Furthermore, in

this kind of communication environments, the com-

plexity of the adopted protocols and the high capac-

ity of the network infrastructure bring the need to de-

velop and employ new processing strategies for this

kind of Big Data. At this purpose, this ﬁeld has been

deeply studied using computational intelligence ap-

proaches over the last two decades (Sperotto et al.,

2010; Bhuyan et al., 2014) following the new tech-

nological innovation in this area, from IoT (Roux

et al., 2018) to Software Deﬁned Networking (Abhi-

https://orcid.org/0000-0001-8014-9152

https://orcid.org/0000-0003-1730-5436

https://orcid.org/0000-0003-4391-2598

https://orcid.org/0000-0001-8244-0015

lash and Divyansh, 2018). The importance gained by

the IoT cybersecurity research ﬁeld has furthermore

highlighted the importance of building Network In-

trusion Detection Systems (NIDSs) capable of work-

ing in an environment made of mobile heterogeneous

devices, connected by the IEEE 802.11 standard (Wi-

Fi). So, extending the focus also on the Physical and

Medium Access Control (MAC) levels, rather than

just network, transport or application levels, is needed

to fulﬁl the security gap in IoT and industrial sen-

sors networks (Roux et al., 2018; Anton et al., 2019).

For this purpose, in (Kolias et al., 2016) an extensive

study of possible Wi-Fi attacks in a real Small Ofﬁce

Home Ofﬁce (SOHO) environment has been carried

out, which included multiple workstations and smart

devices. The major contribution proposed in (Kolias

et al., 2016) is the publication of the Aegean Wi-Fi

Intrusion Detection (AWID) dataset, able to work as

an interesting test bed for new processing strategies

for network anomaly and intrusion detection.

In this work, we further study and develop com-

putational intelligence tools starting from a previous

recent work (Rizzi et al., 2020), in order to provide

advanced approaches for network attack classiﬁcation

412

Granato, G., Martino, A., Baldini, L. and Rizzi, A.

Intrusion Detection in Wi-Fi Networks by Modular and Optimized Ensemble of Classiﬁers.

DOI: 10.5220/0010109604120422

In Proceedings of the 12th International Joint Conference on Computational Intelligence (IJCCI 2020), pages 412-422

ISBN: 978-989-758-475-6

and automatic knowledge discovery. The key points

that we target in this work are:

• an improved modular classiﬁcation system able to

be automatically adapted to detect multiple kind

of attacks, potentially not exclusively in Wi-Fi

networks;

• an improved processing methodology with further

automatic knowledge discovery capabilities at the

training stage;

• a further study on different approaches to detect

and classify attacks based on single frame analy-

sis, hopefully to employ relatively cheap hardware

to gain maximum operational advantage.

Our proposed processing architecture is based on an

array of simple two-class classiﬁers, each one spe-

cialized in discriminating frames speciﬁcally forged

to be part of a speciﬁc attack against all other types

of frames. We further enhance the knowledge discov-

ery capabilities of the training phase by adopting a

genetic meta-heuristic procedure that not only selects

relevant features, but also tunes the hyper-parameters

of the employed clustering procedure and classiﬁca-

tion algorithm in order to maximize the classiﬁcation

accuracy and the knowledge discovery capabilities.

Lastly, we show that a remarkable accuracy can be

obtained in spotting malicious frames, while main-

taining a low false alarm rate. In particular, out of 14

attack classes, the proposed strategy is able to achieve

an accuracy level greater than 90% for 10 of them and

99% for 7 of them.

The rest of the paper is organized as follows. Af-

ter reviewing the related literature in Section 2, we

present the main characteristics of the AWID dataset

and outline the attacks in Section 3, along with the

pre-processing stage and the deﬁnition of the dissim-

ilarity measure adopted to quantify the dissimilarity

between data packets. An account of the machine

learning algorithms used for our system is given in

Section 4. The design of Wi-Fi attacks classiﬁer is

outlined in Section 5. The experimental results are

presented in Section 6. Finally, conclusions and fu-

ture works are drawn in Section 7.

2 RELATED WORKS

The Big Data problem deﬁned in the previous Sec-

tion has been deeply studied using machine learning

techniques as a mean to develop automatic anomaly

detection and attack classiﬁcation tools. One of the

key advantages of machine learning techniques over

traditional approaches is that they do not demand

pre-made signatures of attack frames (Bhuyan et al.,

2014). In particular, in the ﬁeld of IoT and enter-

prise Wi-Fi networks, multiple approaches have been

studied so far. Starting with the work of Kolias et al.

(Kolias et al., 2016), a deep study of Wi-Fi attacks in

WEP protected networks has been carried out in or-

der to correctly build a reference dataset to study dif-

ferent approaches to the intrusion detection problem.

The authors present the ﬁrst attempt to perform net-

work trafﬁc classiﬁcation over the AWID dataset us-

ing machine learning techniques provided by the open

source software WEKA (Frank et al., 2016). Further-

more, authors provide a ﬁrst analysis of key features

required for the classiﬁcation problem.

In (Benza

ıd et al., 2016) network trafﬁc has been

analyzed at MAC level using artiﬁcial neural net-

works, suitably trained to detect address spooﬁng at-

tacks.

In (Guennoun et al., 2008; El-Khatib, 2010) an

approach based on k-means clustering and multilayer

perceptrons has been studied, used jointly with infor-

mation gain methods, achieving accuracy near 90%.

The dataset used in those works is not public.

Multiple machine learning algorithms have been

employed in (Agarwal et al., 2015) in order to detect

de-authentication Denial of Service attacks.

Particular focus has been given to the evil twin

attack and fake Access Point (AP) detection in (Taka-

hashi et al., 2010) and (Lanze et al., 2014), in order

to ﬁnd anomalous network trafﬁc using ﬁngerprint-

ing techniques and to distinguish (real) hardware APs

from (fake) software ones.

In (Thing, 2017) a deep learning approach for

multi-class classiﬁcation has been considered in order

to detect macro-classes of network trafﬁc (legitimate

trafﬁc, ﬂooding, injection and impersonation attacks),

along with feature self-learning strategies by means

of a deep learning approach based on a stacked auto-

encoder with different activation functions.

In (Aminanto and Kim, 2017), the authors focus

on improving feature selection and detection of the

impersonation attacks by using a deep learning ap-

proach based on a stacked auto-encoder algorithm on

the AWID dataset. They exploit artiﬁcial neural net-

works to perform feature selection adopting a manual

threshold applied to the ﬁrst hidden layer of an arti-

ﬁcial neural network and a stacked auto-encoder ap-

proach to classify patterns.

In (Kolias et al., 2017), ant colony optimization

has been employed on a pre-processed AWID dataset,

aggregating features to preserve privacy in a central

node and ﬁnd new ways to detect attacks. The au-

thors perform off-line trafﬁc analysis by adopting a

MAC cumulative statistics approach studying frames

in pre-deﬁned time windows. Effectiveness of the de-

Intrusion Detection in Wi-Fi Networks by Modular and Optimized Ensemble of Classiﬁers

413

veloped algorithm is demonstrated in particular by the

interesting IF-THEN rule-based interpretability point

of view, for which has been described how rich of

information could be frames’ time windows to ﬁnd

malicious frames.

In (Qin et al., 2018), the authors proposed an in-

trusion detection study based on a custom manual pre-

processing scheme based on the calculation of lin-

ear dependencies between features, with a features

selection stage using a two dimensional data clean-

ing approach. Finally, Support Vector Machines with

Gaussian radial basis function kernel are employed

for classiﬁcation. This classiﬁcation scheme realizes

an ensemble of binary classiﬁers in order to solve the

multi-class classiﬁcation problem. In this work, the

authors used the AWID dataset by adopting the same

4-way labelling already used in (Thing, 2017).

Combined use of Kernel Density Estimation and

Hidden Markov Models through a tandem queueing

network model has been exploited in (Sethuraman

et al., 2019). Authors pre-processed a subset of the

AWID dataset, selecting features using a probabilis-

tic approach in order to obtain a network ﬂows set

of patterns. The trained model is capable of obtain-

ing interesting performance. The interpretability of

computational intelligence models is the core of this

work, which stems from a previous study (Rizzi et al.,

2020). In the latter, we also exploited patterns’ fea-

tures extracted from a single MAC frame in order to

check its maliciousness, and associate it to speciﬁc

attack strategies. In this work, we further propose:

• a revised optimization process based on a new ﬁt-

ness function which jointly considers a suitably

deﬁned penalty function and the Youden’s J statis-

tic (also, informedness) as performance values;

• a new genetic code for the optimization proce-

dure, which includes the hyper-parameters for

both clustering and classiﬁcation stages in order

to select, for each attack class, the best training

set resolution and the best setup parameters;

• a second lightweight genetic algorithm able to

adapt thresholds for false alarm management.

3 DATASET DESCRIPTION

In this work we make use of the AWID dataset to

check performance and evaluate the validity of the

proposed system. This dataset has been built by

capturing heterogeneous Wi-Fi trafﬁc in a SOHO

network environment realized with multiple devices

(workstations, notebooks, smartphones, smart TVs),

along with a monitor node that passively captures net-

work trafﬁc and an attack node equipped with Kali

Linux. The networking environment where attacks

are carried out consists of infra-structured Wi-Fi net-

works, i.e., Wi-Fi networks where stations commu-

nicate with an AP. The AP issues periodically a so-

called beacon message, announcing itself and a num-

ber of parameters and attributes useful for coordinat-

ing the access to the wireless channel. The whole net-

work is protected by the WEP security protocol. WEP

has been marked as obsolete due to various security

ﬂaws, and replaced by the protocols speciﬁed in the

IEEE 802.11i standard. However, WEP has been cho-

sen for the AWID dataset to simplify the data set ac-

quisition and the construction of the ground truth in-

formation for intrusion detection system experimenta-

tion. Moreover, it is interesting to include attacks on

WEP to test the ability of the machine learning trafﬁc

analyzer to detect those attacks as “anomalies” with

respect to plain network trafﬁc. Indeed, some of the

attacks that have been carried out in the test bed net-

work can be used against WPA2 networks as well (see

Section 3.2). Regular trafﬁc (that is, not affected by

any attack) has been generated by common applica-

tions, such as web browsing, ﬁle transfer, audio/video

streaming. Attacks have been realized by means of

the state-of-the-art aircrack-ng suite

, MDK3

and

other ad-hoc tools. Network trafﬁc belonging to each

attack has been labeled with the related class.

The dataset used in this work is composed of three

parts:

• a single trafﬁc trace which contains examples of

all attack classes, that we take to deﬁne the raw

training set S

;

• twelve trace ﬁles containing legitimate (‘normal’)

trafﬁc, that we use as a ﬁrst test set (S

ts1

);

• twelve trace ﬁles containing mixed normal and at-

tack trafﬁc, that we use as a second test set, (S

ts2

The rationale behind this choice consists in the possi-

bility of highlighting classiﬁers’ performance in two

scenarios: one where no attack is in progress (only

normal trafﬁc), against a second case where a mixed

situation is realized, with normal trafﬁc interleaved

with attack trafﬁc, belonging to different attack types.

A concise description of the attack macro-classes in

the AWID dataset is outlined in Sections 3.1–3.3

with Table 1 summarizing the patterns distribution

for each class within the three different sets, whereas

in Sections 3.4–3.5 we describe the pre-processing

phase and the adopted dissimilarity measure between

MAC frames.

https://www.aircrack-ng.org

https://tools.kali.org/wireless-attacks/mdk3

Further details can be found in (Kolias et al., 2016).

NCTA 2020 - 12th International Conference on Neural Computation Theory and Applications

414

3.1 Flooding

This kind of attacks exploits the lack of authentica-

tion and integrity check in control and management

frames (see (IEEE, 2016) for the detailed deﬁnition

of frame classes) that leads to frames with multi-

ple malleable ﬁelds. The strategy used to modify

frame’s ﬁelds characterize the speciﬁc attack and, de-

pending on the toolchain used, could lead to recog-

nisable values, for instance in terms of sequence

number, reason code or received signal strength.

Eight classes belong to the ﬂooding family: bea-

con, de-authentication, disassociation, amok,

power saving, probe request, rts and cts.

3.2 Frame Injection

This family of attacks aims to forge frames exploiting

vulnerabilities of the WEP security protocol. Forged

frames could be used, for instance, to solicit responses

by victim stations in order to collect cryptographic

material exploitable to recover the keystream, or us-

ing the AP as an oracle and decrypt frames. This kind

of attacks are made possible thanks to security ﬂaws

of WEP, which basically are linked to the malleabil-

ity of the enciphering and a lack of a real message

authentication code. Arp, chop chop and fragmen-

tation belong to this family.

3.3 Impersonation

Impersonation attacks make use of multiple of the

previously described vulnerabilities in order to setup

fake APs as a starting point for more complex attacks.

For instance, these attacks can adopt techniques typ-

ical of arp and fragmentation attacks to collect

cryptographic material. This family sees evil twin,

cafe latte and hirte attacks.

3.4 Pre-Processing

In this work we adopted a subset of the AWID dataset

in order to: suitably train the classiﬁcation system

), check its capabilities in term of false alarms

on a dedicated set made entirely of normal network

trafﬁc (S

ts1

), and verify the accuracy on a mixed at-

tacks case (S

ts2

). The training set has been further

split in order to obtain a validation set S

val

useful

for optimizing performance indices during the train-

ing phase. The applied pre-processing phase has been

split up in two parts: feature engineering and feature

normalization. The former aims to select the most

useful features from the raw data, whereas the lat-

ter is employed to avoid implicit weighting phenom-

Table 1: Patterns per class distribution in the three sets.

Class name S

ts1

ts2

normal 530785 35158851 47325477

amok 477 0 3856

arp 13644 0 500823

beacon 599 0 5498

cafe latte 379 0 16719

chopchop 2871 0 22879

cts 1759 0 38359

deauthentication 4445 0 33870

disassociation 84 0 34871

evil twin 611 0 27045

fragmentation 167 0 240

hirte 19089 0 433750

power saving 165 0 13551

probe request 369 0 10981

rts 199 0 13536

Total 565643 35158851 48481455

ena. Starting from the 156 attributes

(e.g., MAC ad-

dresses, header ﬂags, timestamp, ...) composing each

pattern in the AWID dataset, we have carefully cho-

sen a subset of 25, considered relevant for the purpose

of network intrusion detection following the same ra-

tionale behind our previous work (Rizzi et al., 2020).

This starting set of features are summarized in Ta-

ble 2, and are composed by either numerical, nominal

or boolean types. All considered numerical features

range between 0 and a maximum value that can be

derived from the IEEE 802.11 speciﬁcations (IEEE,

2016), hence each numerical feature f is normalized

as f

(norm)

= f / f

max

3.5 Dissimilarity Measure

Designing an appropriate dissimilarity measure for

the problem and data at hand is a key facet for the suc-

cess of any pattern recognition system. As for Table 2,

we consider patterns made up of numerical (integer-

or real-valued), discrete nominal and boolean fea-

tures. Numerical features are assumed to be normal-

ized in [0, 1]. In this work, we adopt the following

dissimilarity measure, ad-hoc tailored to the hetero-

geneous pattern structure: let x and y be two generic

patterns, then if the i

feature is numerical we let

) = |x

−y

| (1)

whereas if the i

feature is nominal or boolean, we let

) =

(

1 x

6= y

0 x

= y

(2)

http://icsdweb.aegean.gr/awid/

Intrusion Detection in Wi-Fi Networks by Modular and Optimized Ensemble of Classiﬁers

415

Table 2: List of considered features.

Attribute name Data type Description

frame.len Numerical Frame length

radiotap.dbm antsignal Numerical Received Signal Strength

wlan.fc.type Nominal Frame Type

wlan.fc.subtype Nominal Frame Subtype

wlan.fc.ds Nominal Distribution System status

wlan.fc.frag Boolean More Fragments

wlan.fc.pwrmgt Boolean Power management

wlan.fc.order Boolean Order ﬂag

wlan.duration Numerical Duration

wlan.ra Nominal Receiver address

wlan.da Nominal Destination address

wlan.ta Nominal Transmitter address

wlan.sa Nominal Source address

wlan.bssid Nominal BSS ID

wlan.frag Numerical Fragment number

wlan.seq Numerical Sequence number

wlan.fcs good Boolean FCS correctness

wlan mgt.ﬁxed.listen ival Numerical Listen Interval

wlan mgt.ﬁxed.timestamp Numerical Timestamp

wlan mgt.ﬁxed.beacon Numerical Beacon Interval

wlan mgt.ﬁxed.reason code Nominal Reason code

wlan mgt.ﬁxed.sequence Numerical Starting Sequence Number

wlan.wep.iv Nominal Initialization Vector

wlan.wep.icv Nominal Integrity Check Value

data.len Numerical Data Length

Class label String Ground truth metadata

Finally, if we allow each feature to be weighted in-

dependently by means of a binary weighting vector

w ∈ {0, 1}

, the overall dissimilarity measure reads

as:

d(x, y) =

∑

i=1

·d

) (3)

where n is the number of considered features. Con-

sistently with the deﬁnitions given above, the dissim-

ilarity measure takes values in range [0, 1].

4 ADOPTED CLASSIFICATION

SYSTEM

The computational intelligence approach proposed in

this work is composed by the following key compo-

nents, which will be independently described in Sec-

tions 4.1–4.3:

• a granulation strategy: in order to reduce the car-

dinality of the training set while maintaining use-

ful information, we employ a clustering algorithm

that groups patterns in clusters, each one identi-

ﬁed by its representative pattern;

• a classiﬁcation algorithm: which maps patterns

with labels according to a given strategy;

• a genetic optimization procedure: used to ﬁnd a

subset of features per class and an optimal choice

of hyper-parameters for both the granulation strat-

egy and the classiﬁcation algorithm.

4.1 Information Granulation by

Clustering

The training set includes redundant information, mak-

ing it difﬁcult to train algorithms on it without per-

formance issues. At this purpose, we introduce an

information granulation strategy in order to reduce

the cardinality of the training data and to simplify the

training phase. We deﬁne an information granule as a

collection of entities arranged together thanks to their

similarity. So, from this point of view, a cluster, seen

as a set of similar patterns, is a typical example of in-

formation granule (Baldini et al., 2019). Thanks to

a clustering algorithm and by representing each clus-

ter according to a unique representative pattern, it is

possible to perform an information compression to re-

duce the cardinality of a data set, whilst preserving

most of the useful information.

To this aim, we adopted the well-known Basic Se-

quential Algorithmic Scheme (BSAS) (Theodoridis

and Koutroumbas, 2008), a free clustering procedure

where the number of clusters to ﬁnd is not given a-

priori, being an output of the clustering procedure it-

self. The BSAS algorithm depends on the scale pa-

rameter θ, by which it is possible to set the resolu-

tion at which the dataset is analysed that, in turn, is

directly related to the compression ratio. Small val-

ues of θ yield a great number of small clusters, while

higher values for θ return fewer and larger clusters.

Basically, the BSAS algorithm scans the entire set

and alternatively assign each pattern to an exist-

ing cluster or to a newly initiated one, depending on

whether the distance with respect to already existing

clusters is greater than or less than θ. After all patterns

in S

have been scanned, a set of N

clusters emerges.

For each cluster C , its representative is set by the Min-

imum Sum of Distances (MinSoD) criterion (Martino

et al., 2017; Martino et al., 2019b), hence we select

the pattern y ∈ C that minimizes the sum

∑

x∈C

d(x, y),

where d(·,·) is the dissimilarity measure. Hereinafter,

let us denote S

the compressed version of S

4.2 K-Nearest Neighbours

The classiﬁcation phase has been carried out using

the K-Nearest Neighbours (K-NN) (Cover and Hart,

1967) decision rule. Despite its limitations, it is eas-

ily customizable by means of a suitably deﬁned dis-

similarity measure, possibly tailored to work with the

structured data at hand, as in this case.

The K-NN algorithm is deﬁned by the set of

known patterns S

, an integer K and a dissimilarity

measure d(·, ·). Then, given a pattern to be classi-

ﬁed, the label associated to it is assigned by consid-

NCTA 2020 - 12th International Conference on Neural Computation Theory and Applications

416

ering the most frequent labels among its K closest

neighbours according to d(·,·). Note that K-NN, in

its plain version, does not include any training phase.

The set of patterns used to evaluate the nearest neigh-

bours is directly the set of patterns belonging to the

(compressed) training set S

. Hence, the complexity

of this decision rule is given by the cardinality of the

training data (i.e., N

= |S

|). In spite of its simplic-

ity, K-NN is an effective tool, usually providing good

performances, once a suitable dissimilarity measure

has been deﬁned, at the cost of using the entire train-

ing set as a classiﬁcation model. In our case, it is safe

to say that the use of the K-NN rule is strictly related

to the clustering procedure used to granulate the train-

ing set and to compress information.

4.3 Genetic Algorithm

Patterns in the (compressed) training set are charac-

terized by 25 features, each related to a given weight,

to which we add the granulation parameter and the

classiﬁcation algorithm hyper-parameters. An ex-

haustive search in this space is obviously unfeasible

and an effective way to face an automatic feature se-

lection problem consists in adopting evolutionary op-

timization procedures, such as a genetic algorithm

(Goldberg, 1989), with the ﬁnal goal of optimizing a

given ﬁtness function. Each solution in the admissible

domain is represented by a data structure (called ge-

netic code) composed by 27 variables, summarized in

Table 3. Let G denote the size of the population (i.e.,

number of individuals), with the ﬁrst population be-

ing randomly generated. The ﬁtness of each individ-

ual is evaluated and the population is properly sorted

according to their ﬁtness values. At each iteration, a

new generation is constructed according to standard

operators:

Elitism. best individuals are copied to the next gen-

eration;

Selection. s individuals are selected for mating with

probability r;

Crossover. random exchange of genes between the s

individuals previously selected;

Mutation. random ﬂipping of genes;

Immigration. remaining G −([αG] + s) individuals

are randomly deﬁned in the next generation.

The evolutionary optimization is stopped when the

improvement of the ﬁtness of the best individual be-

comes marginal. In our implementation, we set G =

50, α = 0.1, r = 0.2, q = 0.7.

5 ENSEMBLE OF BINARY

CLASSIFIERS FOR WI-FI

ATTACKS DETECTION

5.1 Training the System

The proposed architecture is sketched in Fig. 1. The

whole classiﬁcation system is based on an ensemble

of binary classiﬁers, each one specialized in detect-

ing a given class. For this purpose, each classiﬁer is

individually optimized by means of a ﬁrst genetic al-

gorithm (Section 4.3) in order to ﬁnd suitable hyper-

parameters (K, θ) and suitable relevant features (w,

see Eq. (3)) for each of the problem-related classes.

A second genetic algorithm will be responsible to op-

timize recognition thresholds to ﬁnally decide which

label has to be assigned to each pattern (Section 5.2).

This approach is useful in all those cases in which we

have multi-class classiﬁcation problems with overlap-

ping decision regions that make patterns classiﬁcation

difﬁcult. Furthermore, this allows to carry knowledge

discovery on a class-aware basis, allowing to select a

tailored subset of useful features for each attack class.

(θ

, w

, S

tr(1)

)

(θ

, w

, S

tr(p)

)

(ω

, r

)

(ω

, r

)

Frames

in input

Figure 1: Proposed architecture: each model M

, related

to the i

class, processes the input frame by considering

the weighting vector w

and the custom training set S

tr(i)

obtained through the granulation strategy setup with θ

Table 3: Genetic code description.

Parameter Range

Max cluster radius θ [0,0.1]

Number of neighbours K [1,21]

Feature selector w

{0,1}

The ﬁtness function of the genetic algorithm is

made of two main contributions: the informedness

(see Eq. (4)), a summary performance index, and the

penalty (see Eq. (5)), a correction factor useful to

avoid an extreme training set compression (i.e., risk

of removing useful information). The informedness

index represents the probability of an informed de-

cision, giving equal importance to false positive and

false negative values (Powers, 2011). Different mea-

surements that gives the same index value have the

same proportion of total misclassiﬁed results. This

Intrusion Detection in Wi-Fi Networks by Modular and Optimized Ensemble of Classiﬁers

417

Figure 2: Proposed processing architecture for the training phase.

performance index is useful in operational contexts

where we have the vast majority of patterns repre-

senting normal trafﬁc packets, so the high false alarm

rate could imply an heavy loss of network through-

put. Formally, we can introduce the informedness in-

dex (Youden, 1950), normalized as in (Martino et al.,

2019a; Martino et al., 2020a; Martino et al., 2020b),

with the following equation:

J =

+ S

, J ∈ [0, 1] (4)

where S

and S

indicate sensitivity and speciﬁcity,

respectively. The penalty index, as instead, has been

introduced in order to effectively perform training set

compression without loosing excessive information.

This index will be signiﬁcantly high in case the clus-

tering algorithm returns a number of representatives

for a given class lower than 2. Furthermore, the num-

ber of neighbours K must be chosen compatibly with

the quantity of representatives found. So we assign

a penalty value to each single class outcome of the

clustering algorithm applied to the training set, the

mean of that values will be subtracted from the clas-

siﬁcation performance within the ﬁtness function for

each individual of the genetic algorithm. Mathemati-

cally, the penalty function is described by the follow-

ing equation:

P =

∑

i=1

−(m

(θ

)−1)

, P ∈[0,1] (5)

where p is the number of classes, and m

(θ

) is

the number of clusters found for class i, which de-

pends on the θ

parameter for the granulation strategy.

Hence the ﬁtness function, to be maximized on S

val

reads as:

f = J(w

) −P(θ

), f ∈ [0,1] (6)

5.2 Contentions Resolution

Each binary One-Against-All classiﬁer belonging to

the ensemble detects if a given pattern belongs to the

target class, so it could happen that multiple classi-

ﬁers mark with different labels the same Wi-Fi frame.

In order to properly evaluate the correctness of each

classiﬁer, we adopt a Winner-Takes-All (WTA) strat-

egy based on the reliability measure analysis of the

output labels of each classiﬁer. This analysis is per-

formed using a simple multi-class K-NN decision

rule, that processes each frame by considering the

output of each classiﬁer. So each classiﬁer, for each

pattern, returns the predicted label and the reliability

measure of its prediction. Mathematically, we used

the following deﬁnition as reliability measure:

R =

∑

∀i∈W

(1 −d

) −

∑

∀i∈S

(1 −d

) −



−

−1



1 −



−

−1



(7)

where W and S are the sets of neighbours belong-

ing to the ﬁrst and second most popular class, respec-

tively. This kind of measure:

• considers the purity of the neighbours set (i.e., the

number of patterns of the winner class) and their

distances d

with respect to the test pattern;

• considers the number of patterns of the looser

class and their corresponding distances from the

classiﬁed pattern;

• lastly, takes values in the range [0,1].

Since we are adopting a feature selection mechanism,

each classiﬁer will likely work on a different subspace

of the dataset. In order to fairly compare their outputs

in the WTA, we performed a further normalization of

the dissimilarity measure (formerly Eq. (3)). Hence,

for the j

classiﬁer, we have:

(x,y) =

√

∑

i=1

·d

) (8)

where n

indicates the number of selected features by

the j

classiﬁer.

5.3 False Alarm Management

NIDSs’ actions have an impact on the Quality of Ser-

vice of the network. One of the key parameters that let

us to evaluate NIDSs activity during attack prevention

is the false alarm rate. An excessive number of false

NCTA 2020 - 12th International Conference on Neural Computation Theory and Applications

418

alarms raised by the NIDS can be harmful both in the

cases of suspicious frames dropping and, from a secu-

rity analyst viewpoint, might lead to handling a large

volume of false alarms that need a proper counterac-

tion, according to the security policy. In this work, we

studied a solution able to reduce this problem based

on a dedicated binary classiﬁer which targets the nor-

mal class and the evaluation of a reliability measure

for binary classiﬁers outputs (see Section 5.2). Nor-

mal network trafﬁc classiﬁcation is expected to be a

difﬁcult task, since in the same set we consider highly

heterogeneous frames (for example, think about the

legit management and control frames or the dynamics

of the exchange of data frames between stations, de-

pending on the kind of network layer trafﬁc brought).

At the same time, introducing an estimate of trafﬁc

normality can add an important information to use

during the execution of the WTA rule. So, we basi-

cally add a new contender during this phase.

The false alarm management phase is split in three

steps:

1. introduce a dedicated binary classiﬁer which tar-

gets normal frames;

2. apply a threshold τ on the reliability measure of

the output label of each classiﬁer;

3. optimize these thresholds using a genetic opti-

mization procedure in order to let the ensemble

of classiﬁers to take decisions about labels.

This genetic procedure is based on the same standard

principles of the previous one, yet the genetic code

is deﬁned as the vector of reliability thresholds as re-

turned by the classiﬁer ensemble. The ﬁtness corre-

sponds to the precision of each classiﬁer.

6 TESTS AND RESULTS

At the end of the training procedure, the genetic algo-

rithm returns, for each label, an optimal set of features

and a suitable set of hyper-parameters for both clus-

tering and the classiﬁcation algorithms. Conversely,

the second genetic optimization phase adapts relia-

bility measure thresholds to improve perfomance and

false alarms management. In Table 4 we summarize

the genetic codes found during the genetic optimiza-

tion procedures (K, θ, w and τ), along with other in-

dices such as informedness, precision, best ﬁtness and

cardinality of the compressed training set, in order to

let us analyze results from different perspectives. In

Table 5 we have resumed results obtained using both

test sets S

ts1

and S

ts2

As we can see, performance are interesting and we

can make some considerations. First of all, we con-

ﬁrm that almost all ﬂooding attacks are easily iden-

tiﬁable on a per-frame basis. This is particularly true

for all those attacks based on management frames, for

which we can provide a precise feature set. As shown

in Table 4, we can ﬁnd all key features automatically

selected by this system for this type of attacks (e.g.,

type, subtype, reason code). After all, this conclu-

sion is not true for ﬂooding attacks based on control

frames like rts and cts. In fact, these attacks exploit

frames used for MAC, so they are very simple and not

identiﬁable without taking in consideration an analy-

sis based on a suitably ordered set of frames. Perfor-

mance indexes obtained on a mixed attack case (S

ts2

)

are conﬁrmed also in a normal-only situation (S

ts1

demonstrating low false alarm rate (i.e., 1’s comple-

ment of the reported accuracy) capabilities for the

trained classiﬁcation system.

Secondly, concerning attacks that aim to decrypt

the payload or retrieve the keystream itself, results are

similar to ﬂooding attacks. When this kind of attacks

become part of more complex attack vectors, classiﬁ-

cation performance drop since a ﬂow-based analysis

approach is required.

The introduction of the reliability measure thresh-

old based on the normal One-Against-All classiﬁer,

allows us to maintain low false alarm rates, paying the

price of lower classiﬁcation accuracy since the rule to

assign an attack is much more severe.

Lastly, impersonation attacks are the most difﬁcult

to classify due to the extremely similar behaviour to a

normal scenario. The accuracy obtained in these cases

is low and highly variable depending on the particular

test set.

An almost common key feature automatically se-

lected by the classiﬁcation system during the training

phase is the received signal power strength, which is

measured by the wireless receiver of the monitor sta-

tion. In this manner, the training procedure is able to

take in consideration the spatial position of the trans-

mitter station, embedded in the received signal power

strength. On a single frame classiﬁcation problem,

this could be the best feature able to help detect ﬂood-

ing attacks based on rts and cts control frames, (ex-

cept for the classic MAC addresses dictionary, which

is not easily scalable).

We conclude this analysis by comparing, as much

as possible, results obtained in this work against other

recent works (see Table 6) on the same dataset, to the

best of our knowledge. Since those works employ

different AWID subsets, different class labels subsets

and different objective functions, performance com-

parison is only possible to a limited extent.

Methodologically speaking, one of the most im-

portant aspects developed in our work that others do

Intrusion Detection in Wi-Fi Networks by Modular and Optimized Ensemble of Classiﬁers

419

Table 4: Optimization phase results for each classiﬁer. Features in the rightmost column are ordered as in Table 2.

Class name θ |Str| K Best Fitness J Precision τ Selected Features (w)

amok 0.0029 28 5 0.8238 0.9967 0.8681 0.689 1110001010111111100110110

arp 0.0002 800 1 0.998 0.9985 1 0.934 1111111111111101110110111

beacon 0.0091 263 1 0.7751 0.997 0.9944 0.806 1101001011111111111110111

cafe latte 0.0049 8 1 0.8126 0.994 0.0263 0.75 1111111110111111101011111

chop chop 0.0104 800 1 0.7254 0.9534 0.9977 0.856 0010001110111111101110110

cts 0.0005 15 1 0.9393 0.9489 1 0.96 0111111111111101110010111

deauthentication 0.004 6 11 0.8169 1 0.997 0.045 1110000110001111101011110

disassociation 0.0009 22 5 0.8759 0.9999 1 0.983 0011101011111111100110111

evil twin 0.0047 3 5 0.5054 0.5271 0.0109 0.024 1100000010011111101010110

fragmentation 0.0002 107 7 0.8280 0.9971 1 0.656 1011111110001111110010110

hirte 0.0587 51 1 0.5213 0.9861 0.9923 0.732 1101111101111110100110111

power saving 0.0029 10 1 0.8636 0.9907 1 0.988 0100111100111110111010110

probe request 0.0028 2 1 0.8272 0.9999 1 0.91 1101100101111111110011111

rts 0.0002 6 1 0.9983 0.9988 1 0.965 1111111111111111111111111

Table 5: Mean accuracy obtained on the two considered test

sets.

Class name

ts1

Mean (Std)

ts2

Mean (Std)

amok 0.9999 (0.0002) 0.9999 (1.25e-07)

arp 0.9270 (0.0634) 0.9176 (0.0042)

beacon 0.9825 (0.0131) 0.9989 (4.77e-07)

cafe latte 0.7911 (0.1092) 0.9169 (0.0012)

chop chop 0.9895 (0.0112) 0.9899 (0.0001)

cts 0.7582 (0.171) 0.5177 (0.0031)

deauthentication 1 (0) 0.9998 (5.02e-07)

disassociation 1 (0) 0.9997 (4.98e-10)

evil twin 1 (0) 0.9993 (1.98e-06)

fragmentation 1 (0) 0.9997 (2.66e-07)

hirte 0.3205 (0.1811) 0.7015 (2.9e-07)

power saving 0.6966 (0.1275) 0.8689 (0.0042)

probe request 1 (0) 0.9999 (5e-10)

rts 0.9599 (0.0576) 0.844 (0.0031)

not employ is an automatic algorithm setup and fea-

tures selection strategy based on a genetic algorithm

which detects relevant features in an attack-aware

fashion, returning interpretable information (genetic

codes). Furthermore, we employed two different sub-

set of the AWID dataset in order to perform an ex-

plicit evaluation of our system for false alarms man-

agement. Other than numerical results, the following

aspects hold: in (Thing, 2017), the overall accuracy is

greater than 98%, but with some caveats considering

the ﬂooding attack class. Furthermore, they employ

an embedded self-learning approach via a deep neu-

ral network for feature selection, yet do not provide

an in-depth analysis. Results in (Aminanto and Kim,

2017) are rather comparable with the proposed ones.

In (Kolias et al., 2017), despite achieved accuracy is

77.8%, the output of the classiﬁcation algorithm in-

cludes human-readable IF-THEN rules to perform ﬁl-

tering activities. In our case, we do not provide ex-

plicit rules, but we ﬁnd key features of each attack in

order to try to understand and characterize malicious-

vs-normal Wi-Fi ﬂows. Finally, in (Qin et al., 2018),

the set of interesting attributes found with their tech-

nique is similar to these found in this work. Classi-

ﬁcation accuracy of injection attacks is comparable,

while impersonation and ﬂooding attacks are recog-

nized hardly. Overall, the advantage of the proposed

approach, if compared in particular to deep learning-

based studies, rely on the fact that the obtained ge-

netic codes are easily human-understandable in terms

of key features found, clusters size and classiﬁcation

algorithm setup, and summarize each attack class be-

havior. Furthermore, this approach provides a bet-

ter versatility of the training phase that can be cus-

tomized for each class of the classiﬁcation problem,

making the overall structure modular with respect to

the problem-related classes: in fact, as new attacks

need to be modelled, one can simply add the corre-

sponding One-vs-All classiﬁer to the ensemble, with-

out need to retrain the whole system. This not only

makes the training stage faster, but also makes the

proposed approach appealing towards low-cost hard-

ware.

7 CONCLUSIONS

In this work, we proposed a scalable and modular ar-

chitecture to detect attacks in Wi-Fi networks by an-

alyzing frames using a machine learning approach.

The proposed system is made of an ensemble of One-

Against-All classiﬁers each one targeting a speciﬁc

attack class, suitably trained and optimized using a

NCTA 2020 - 12th International Conference on Neural Computation Theory and Applications

420

Table 6: Performance comparison with related state-of-the-art works.

Related Works Dataset Size Classiﬁcation Problem Method

Feature Selection

Strategy

Performance

Index

Performance (%)

(Kolias et al., 2016) Entire AWID dataset

Both using macro-classes

and single attack labels

Random Tree J48 Manual Accuracy 96.1982

(Aminanto and Kim, 2017)

AWID subset

(∼2 millions patterns)

4 macro-classes Deep Learning Automatic Accuracy

92-98

(depending on class)

(Kolias et al., 2017) Entire AWID dataset

Both using macro-classes

and single attack labels

Ant Colony-based

Both manual

and automatic

Accuracy 77.8

(Thing, 2017)

AWID subset

(∼2.4 millions patterns)

4 macro-classes Deep Learning Automatic Accuracy 98.6688

(Qin et al., 2018)

AWID subset

(∼1.3 millions patterns)

4 macro-classes SVM

Both manual

and automatic

Accuracy

89-99

(depending on class)

(Sethuraman et al., 2019)

Modiﬁed subset

of the AWID dataset

7 attack classes HMM and KDE Automatic Precision 92.28

(Rizzi et al., 2020)

AWID subset

(∼84 millions patterns)

14 attack classes

BSAS and K-NN

driven by GA

Both manual

and automatic

Accuracy

92.446

(false alarm rate 12.1)

This work

AWID subset

(∼84 millions patterns)

14 attack classes

BSAS and K-NN

driven by GA

Both manual

and automatic

Accuracy

91.1

(false alarm rate 5.9)

genetic algorithm.

This approach let us simplify the classiﬁcation

problem by maximizing the training capabilities with

a smaller training set, leading to a reduced amount of

dissimilarity measure computations between patterns.

During the training phase, we optimize the granula-

tion strategy hyper-parameters (θ, in this case) for

each attack class, creating optimized training subset

of patterns. The very same optimization procedure

performs features selection in order to simplify data

processing by each classiﬁer and to extract knowl-

edge about recurrent characteristics of each attack.

Correctness of knowledge discovery capabilities have

been evaluated by comparing results with attack tools

output (see Section 6). Furthermore, the proposed ap-

proach has been designed in order to have a natively

parallel design, easily employable on multi-threaded

architectures on general purpose multi-core systems,

or on a dedicated hardware implementation on FPGA

or ASIC circuits.

Intrusion detection is performed by analyzing net-

work trafﬁc frame-by-frame, obtaining performance

values that show an average of more then 90% of

frames correctly classiﬁed, despite of the difﬁculties

to model the complex normal trafﬁc process. With

respect to our previous system performances (Rizzi

et al., 2020), we have obtained comparable results in

terms of accuracy, while greatly improving the false

alarm rate (see Table 6). These results are comparable

with state-of-the-art works whilst providing the fur-

ther advantages explained before. To further improve

the reliability of the classiﬁer and the knowledge dis-

covery capabilities, in particular for those attacks that

do not allow single frame analysis with such a high

success rate, higher complexity tools should be de-

veloped, that look through sequences of frames. In

future works we will study the impact of strategies

that let us represent Wi-Fi patterns in different math-

ematical spaces, able to enhance machine learning al-

gorithms capabilities to classify and cluster patterns

whilst maintaining white-box models that, if com-

bined with automatic features selection methods, are

able to output human readable ﬁltering rules. Prelim-

inary works show that a subset of attacks is recog-

nizable more easily considering sequences of frames

conveniently collected and ordered.

REFERENCES

Abhilash, G. and Divyansh, G. (2018). Intrusion detec-

tion and prevention in software deﬁned networking.

In 2018 IEEE International Conference on Advanced

Networks and Telecommunications Systems (ANTS),

pages 1–4.

Agarwal, M., Biswas, S., and Nandi, S. (2015). Detection

of de-authentication dos attacks in wi-ﬁ networks: A

machine learning approach. In 2015 IEEE Interna-

tional Conference on Systems, Man, and Cybernetics

(SMC), pages 246–251.

Aminanto, M. E. and Kim, K. (2017). Detecting imper-

sonation attack in wiﬁ networks using deep learning

approach. In Choi, D. and Guilley, S., editors, Infor-

mation Security Applications, pages 136–147, Cham.

Springer International Publishing.

Anton, S. D. D., Fraunholz, D., and Schotten, H. D. (2019).

Using temporal and topological features for intrusion

detection in operational networks. In Proceedings

of the 14th International Conference on Availability,

Reliability and Security, ARES ’19, New York, NY,

USA. Association for Computing Machinery.

Baldini, L., Martino, A., and Rizzi, A. (2019). Stochastic

information granules extraction for graph embedding

and classiﬁcation. In Proceedings of the 11th Inter-

national Joint Conference on Computational Intelli-

gence - Volume 1: NCTA, (IJCCI 2019), pages 391–

402. INSTICC, SciTePress.

Benza

ıd, C., Boulgheraif, A., Dahmane, F. Z., Al-Nemrat,

A., and Zeraoulia, K. (2016). Intelligent Detection

of MAC Spooﬁng Attack in 802.11 Network. In

Proceedings of the 17th International Conference on

Intrusion Detection in Wi-Fi Networks by Modular and Optimized Ensemble of Classiﬁers

421

Distributed Computing and Networking, ICDCN ’16,

pages 47:1–47:5, New York, NY, USA. ACM.

Bhuyan, M. H., Bhattacharyya, D. K., and Kalita, J. K.

(2014). Network anomaly detection: Methods, sys-

tems and tools. IEEE Communications Surveys Tuto-

rials, 16(1):303–336.

Cover, T. and Hart, P. (1967). Nearest neighbor pattern clas-

siﬁcation. IEEE transactions on information theory,

13(1):21–27.

El-Khatib, K. (2010). Impact of Feature Reduction on the

Efﬁciency of Wireless Intrusion Detection Systems.

IEEE Transactions on Parallel and Distributed Sys-

tems, 21(8):1143–1149.

Frank, E., Hall, M. A., and Witten, I. H. (2016). Data min-

ing: practical machine learning tools and techniques.

Morgan Kaufmann, 4 edition.

Goldberg, D. E. (1989). Genetic Algorithms in Search, Op-

timization and Machine Learning. Addison-Wesley

Longman Publishing Co., Inc., Boston, MA, USA, 1st

edition.

Guennoun, M., Lbekkouri, A., and El-Khatib, K. (2008).

Selecting the Best Set of Features for Efﬁcient In-

trusion Detection in 802.11 Networks. In Informa-

tion and Communication Technologies: From Theory

to Applications, 2008. ICTTA 2008. 3rd International

Conference on, pages 1–4.

IEEE (2016). Ieee standard for information technol-

ogy—telecommunications and information exchange

between systems local and metropolitan area net-

works—speciﬁc requirements - part 11: Wireless lan

medium access control (mac) and physical layer (phy)

speciﬁcations. IEEE Std 802.11-2016 (Revision of

IEEE Std 802.11-2012), pages 1–3534.

Kolias, C., Kambourakis, G., Stavrou, A., and Gritzalis, S.

(2016). Intrusion detection in 802.11 networks: Em-

pirical evaluation of threats and a public dataset. IEEE

Communications Surveys Tutorials, 18(1):184–208.

Kolias, C., Kolias, V., and Kambourakis, G. (2017). Ter-

mid: A distributed swarm intelligence-based approach

for wireless intrusion detection. Int. J. Inf. Secur.,

16(4):401–416.

Lanze, F., Panchenko, A., Braatz, B., and Engel, T. (2014).

Letting the Puss in Boots Sweat: Detecting Fake Ac-

cess Points Using Dependency of Clock Skews on

Temperature. In Proceedings of the 9th ACM Sympo-

sium on Information, Computer and Communications

Security, ASIA CCS ’14, pages 3–14.

Martino, A., Frattale Mascioli, F. M., and Rizzi, A. (2020a).

On the optimization of embedding spaces via infor-

mation granulation for pattern recognition. In 2020

International Joint Conference on Neural Networks

(IJCNN). Accepted for Publication.

Martino, A., Giuliani, A., and Rizzi, A. (2019a). (hy-

per)graph embedding and classiﬁcation via simplicial

complexes. Algorithms, 12(11).

Martino, A., Giuliani, A., Todde, V., Bizzarri, M., and

Rizzi, A. (2020b). Metabolic networks classiﬁcation

and knowledge discovery by information granulation.

Computational Biology and Chemistry, 84:107187.

Martino, A., Rizzi, A., and Frattale Mascioli, F. M. (2017).

Efﬁcient approaches for solving the large-scale k-

medoids problem. In Proceedings of the 9th Inter-

national Joint Conference on Computational Intelli-

gence - Volume 1: IJCCI,, pages 338–347. INSTICC,

SciTePress.

Martino, A., Rizzi, A., and Frattale Mascioli, F. M.

(2019b). Efﬁcient approaches for solving the large-

scale k-medoids problem: Towards structured data.

In Sabourin, C., Merelo, J. J., Madani, K., and War-

wick, K., editors, Computational Intelligence: 9th In-

ternational Joint Conference, IJCCI 2017 Funchal-

Madeira, Portugal, November 1-3, 2017 Revised Se-

lected Papers, pages 199–219. Springer International

Publishing, Cham.

Powers, D. M. W. (2011). Evaluation: From precision, re-

call and f-measure to roc., informedness, markedness

& correlation. Journal of Machine Learning Tech-

nologies, 2(1):37–63.

Qin, Y., Li, B., Yang, M., and Yan, Z. (2018). Attack detec-

tion for wireless enterprise network: a machine learn-

ing approach. In 2018 IEEE International Conference

on Signal Processing, Communications and Comput-

ing (ICSPCC), pages 1–6.

Rizzi, A., Granato, G., and Baiocchi, A. (2020). Frame-by-

frame wi-ﬁ attack detection algorithm with scalable

and modular machine-learning design. Applied Soft

Computing, 91:106188.

Roux, J., Alata, E., Auriol, G., Ka

aniche, M., Nicomette,

V., and Cayre, R. (2018). Radiot: Radio communi-

cations intrusion detection for iot - a protocol inde-

pendent approach. In 2018 IEEE 17th International

Symposium on Network Computing and Applications

(NCA), pages 1–8.

Sethuraman, S. C., Dhamodaran, S., and Vijayakumar, V.

(2019). Intrusion detection system for detecting wire-

less attacks in ieee 802.11 networks. IET Networks,

8(4):219–232.

Sperotto, A., Schaffrath, G., Sadre, R., Morariu, C., Pras,

A., and Stiller, B. (2010). An Overview of IP Flow-

Based Intrusion Detection. IEEE Communications

Surveys Tutorials, 12(3):343–356.

Takahashi, D., Xiao, Y., Zhang, Y., Chatzimisios, P., and

Chen, H.-H. (2010). IEEE 802.11 User Fingerprinting

and Its Applications for Intrusion Detection. Comput.

Math. Appl., 60(2):307–318.

Theodoridis, S. and Koutroumbas, K. (2008). Pattern

Recognition. Academic Press, 4 edition.

Thing, V. L. L. (2017). IEEE 802.11 Network Anomaly

Detection and Attack Classiﬁcation: A Deep Learning

Approach. In 2017 IEEE Wireless Communications

and Networking Conference (WCNC), pages 1–6.

Youden, W. J. (1950). Index for rating diagnostic tests. Can-

cer, 3(1):32–35.

NCTA 2020 - 12th International Conference on Neural Computation Theory and Applications

422