A NEW FRAMEWORK FOR REAL-TIME ADAPTIVE FUZZY
MONITORING AND CONTROL FOR HUMANS UNDER
PSYCHOPHYSIOLOGICAL STRESS
A. Nassef, C. H. Ting, M. Mahfouf, D. A. Linkens
Department of Automatic Control and Systems, The University of Sheffield, Sheffield, United Kingdom
P. Nickel, G. R. J. Hockey, A. C. Roberts
Department of Psychology, The University of Sheffield, Sheffield, United Kingdom
Keywords: Adaptive Automation, Operator Functional State, Cardiovascular System, Electroencepharograph, Fuzzy
Systems, Genetic Algorithms, Signal Processing.
Abstract: This paper proposes a new framework for the on-line monitoring and adaptive control of
psychophysiological markers relating to humans under stress. The starting point of this framework relates to
the assessment of the so-called operator functional state (OFS) using physical as well as psychological
measures. An adaptive neural-fuzzy model linking Heart-Rate Variability (HRV) and Task Load Index
(TLI) with the subjects’ optimal performance has been elicited and validated via a series of real-life
experiments involving process control tasks simulated on an Automation-Enhanced Cabin Air Management
System (aCAMS). The elicited model has been used as the basis for an on-line control system, whereby the
model predictions which indicate whether the actual system is in error or not, have been used to modify the
level of automation which the system may operates under.
1 INTRODUCTION
With increasingly complex design of automation in
safety-critical applications, there is a growing
concern for the consequences of performance
breakdown. This is because the human operator’s
role has become compromised with increasing
operational demand, stress and fatigue, which all
threaten safety and reliability (Hockey et al., 2003).
The approach taken to this problem in this paper is
based on an ‘Operator Functional State’ (OFS)
framework in which the performance of the operator
is constrained by requirements to manage the
automation tasks and his/her own personal state.
The OFS model should predict that, for a period
before manifest breakdown occurs, the operator will
be in a vulnerable state, because of reduced spare
capacity to respond to emergencies. The goal of the
current programme of work is to develop models for
evaluating psychophysiological markers of this high
risk strain state. If such states can be reliably
detected, they can be used to trigger a switch of
control from human to computer, through an
adaptive automation (AA) interface, reducing the
risk of operational breakdown (Kaber et al. 2001).
A likely marker is the ‘task load index’ (TLI)
identified by Gevins and his group (Gevins and
Smith, 1999). TLI is based on the presence of high
levels of theta activity at frontal midline sites, with
concomitant attenuation of alpha power in parietal
sites [theta/alpha]. Observation of reduced frontal-
midline theta power may reflect direct effects of
fatigue or strategic disengagement from the
executive requirements of the task management
(Lorenz and Parasuraman, 2003).
To investigate this, a task known as automation-
enhanced Cabin Air Management System (aCAMS)
(Figure 1), developed by Hockey and colleagues
(Hockey et al., 1998, Lorenz, 2002) to simulate the
atmospheric environment within a space capsule, is
used. This semi-automatic system required operators
to maintain an appropriate quantity and quality of
breathable air by keeping system parameters
(temperature, humidity, pressure, O
2
, CO
2
) within
320
Nassef A., H. Ting C., Mahfouf M., A. Linkens D., Nickel P., R. J. Hockey G. and C. Roberts A. (2008).
A NEW FRAMEWORK FOR REAL-TIME ADAPTIVE FUZZY MONITORING AND CONTROL FOR HUMANS UNDER PSYCHOPHYSIOLOGICAL
STRESS.
In Proceedings of the First International Conference on Bio-inspired Systems and Signal Processing, pages 320-325
DOI: 10.5220/0001066003200325
Copyright
c
SciTePress
normal ranges (primary task). The operators
interacted with a dynamic visual display that
provides data on system variables and functions via
a range of controls and automation tools; this is a
large mental burden to the operator.
N
2
O
2
CO
2
scrubber
Heater
Cooler
CABIN
Vent
Dehumidifier
Oxygen
CO
2
Nitrogen
Temperature
Pressure
Humidity
Tasks
Technology
Humans
N
2
O
2
CO
2
scrubber
Heater
Cooler
CABIN
Vent
Dehumidifier
Oxygen
CO
2
Nitrogen
Temperature
Pressure
Humidity
Tasks
Technology
Humans
Figure 1: The aCAMS human-machine system.
The main objective of the research work presented
in this paper is to propose a new framework for the
on-line (real-time) monitoring of the human
operator’s performance for breakdown, stress or
fatigure and the adaptive control of the level of
automation. In order to achieve this a model that
describes the input and output relationship between
the psychophysiological measures (e.g.
cardiovascular and EEG activities) and functional
(i.e. cognitive, mental or psychological) states of the
operator in a simulated process control environment
is built first. The model can then be implemented in
an adaptive automation control system to represent a
kernel in OFS estimation. In the present
investigation, the OFSs identification is achieved by
using adaptive fuzzy modelling which requires the
measured psychophysiological and primary task
performance data only. The proposed modelling
approaches are shown by simulation results to be
capable of effectively exploiting the information
contained in the measured physiological and
performance data. By using this model the OFS may
be identified or predicted by monitoring the changes
in the psychophysiological and performance data,
and hence the model output can be used as a bio-
feedback signal in closed-loop automation control.
This paper is organised as follows: Section 2 will
outline the chosen technical paradigm behind the
intelligent systems-based modelling strategy.
Section 3 will present the final models which were
adopted and Section 4 shows how such models can
be included in the real-time framework for
monitoring and adaptive control. Finally, Section 5
will draw some conclusions in relation to this overall
research study.
2 FUZZY MODELLING OF
OPERATOR FUNCTIONAL
STATE (OFS)
For the purpose of modelling fuzzy logic (Zadeh,
1965) was chosen as the main paradigm for
characterising the input/output mappings because of
its tolerance to uncertainties and also for the fact it
can model human perception in a transparent way
without a greater loss in accuracy. As a result, two
types of fuzzy models were constructed and
optimised automatically: one using neural networks
leading to the Artificial Network Fuzzy Inference
System (ANFIS) architecture (Jang, 1993) which
utilises and the other using Genetic Algorithms
(Goldberg, 1989) to estimate the parameters of the
membership functions and the fuzzy rules of a
Mamdani-type structure (Mamdani, 1974). In order
to carry-out this modelling operation successfully it
is important to first specify the variables associated
with this input/output mapping and then carry-out
the real-time experiments (Mahfouf et al., 2006)
which will enable one to collect the input/output
data information as will be explained next.
2.1 Model Inputs and Output
The candidate inputs of the fuzzy model may
include Heart Rate Variability (HRV) and EEG
markers (TLI), which were found to be most
sensitive to the changes in mental workload
((Fehrengerg and Wientjes, 2000);Nickel et al.,
2005; Zhang et al., 2006). The optimal number of
inputs selected from the above candidate inputs was
determined by linear correlation analysis of the
relationship between the input and output data. The
single output of the model is ‘Time in Range’ related
to the primary task performance.
2.2 Data Acquisition and Analysis
The BioSemi® system (Biosemi, the Netherland)
was used for EEG recording at 32 electrode sites
defined by the international 10-20 system (Jasper,
1958). The electrodes were re-referenced to two
linked mastoids. The EEG signal, sampled at a rate
of 2048 Hz, was pre-processed with a band-pass
filter between 1.6 and 25 Hz. The power in the three
bands (i.e., theta, alpha and beta) for each of the
selected electrode sites was calculated. The primary-
task performance data (‘Time in Range’) were
sampled every 1 min.
The heart rate (HR) signal was recorded every 1
s as soon as the aCAMS was started up. HRV
1
is
defined as the average of the 0.1 Hz component
A NEW FRAMEWORK FOR REAL-TIME ADAPTIVE FUZZY MONITORING AND CONTROL FOR HUMANS
UNDER PSYCHOPHYSIOLOGICAL STRESS
321
powers. HRV
2
is defined as the HR variation
coefficient and given by the following expression:
HR
HR
HRV
μ
σ
=
2
(1)
where σ and μ denote the standard deviation and
average of a HR segment of 7.5 min.
The TLI calculated using different EEG band
powers was proposed in (Gevins et al., 1997). The
TLI indices, TLI
1
and TLI
2
used in this paper, are
given as follows:
=
=
zz
z
POCP
AF
Pz
Fz
P
P
TLI
P
P
TLI
,
,
2
,
,
1
α
θ
α
θ
(2)
where
θ
P and
α
P denote the theta- and alpha-band
power, respectively; the EEG frequency bands are
defined in order as: θ, Fz: 6-7 Hz; α, Pz: 10-12 Hz;
θ, AFz: 5-7 Hz; α, CPz: 8-10.5 Hz; α, POz: 10-13.5
Hz; and Fz, Pz, AFz, CPz, and POz are the five EEG
electrode sites on the scalp introduced in the
standard 10-20 system (Jasper, 1958).
3 RESULTS AND DISCUSSIONS
In this simulation the signal data sampling interval
was taken to be 7.5 min and Gaussian MFs were
used for both fuzzy models. The choice of the
candidate input was mainly driven by the value of
the input-output correlation factor (the higher the
better), the training and testing data correlation
factor (the higher the better) and the MSE values of
the training and testing data. As a result, the two
inputs HRV
1
and TLI
2
were selected for both fuzzy
models. The training and testing data set was
obtained from the 1
st
and 2
nd
experimental sessions,
respectively. The ANFIS modelling result for P2 is
shown in Figure 2.
Due to the large differences between the MSE
values of the model output for each subject another
index was introduced to differentiate between
models. This index was named "Error Factor" and is
defined by the ratio between the MSE of the model
output when using the validating data and the MSE
between the training and validating data as shown in
Equ. (3).
chk-Tr
chk-output model
FactorError
MSE
MSE
=
(3)
Using this new index it was found that Subjects
P2, P4, and P10 led to the highest values, i.e. the
worst performing models compared to the other
subjects. So, those subjects' data have been chosen
for the next study. The optimised rules of Mamdani-
type fuzzy model and their weights are illustrated in
Table 1. The optimal MFs and degrees of belief
(rules’ weight) in each rule are identified by using a
GA approach. It is noted that the 1
st
, 2
nd
, 3
rd
, 11
th
,
12
th
, 13
th
, 15
th
and 16
th
rules (see Table 1 in ‘bold’
characters) are less important in terms of the smaller
weights. The comparison of the model output and
desired output is shown in Figure 3 for P2. Figure 4
illustrates the model output when HRV2 and TLI2
are used as inputs
0 2 4 6 8 10 12 14 16 1
8
50
60
70
80
90
100
Time in range (%)
(a) Comparison of ANFIS output and training data (P2)
Training data
Model output
0 2 4 6 8 10 12 14 16 1
8
20
40
60
80
100
Time index (T
s
= 7.5 min)
Time in range (%)
(b) Comparison of ANFIS output and checking data (P2)
Checking data
Model output
Figure 2: ANFIS modelling results for P2; HRV
1
and TLI
2
as inputs.
0 2 4 6 8 10 12 14 16 18
40
60
80
100
Time in range (%)
(a) Comparison between model output and training data
Mamdani model output
Training data
0 2 4 6 8 10 12 14 16 18
20
40
60
80
100
Time index (T
s
=7.5 min)
Time in range (%)
(b) Comparison between model output and checking data
Mamdani model output
Checking data
Figure 3: Modelling results via the GA-based Mamdani-
type model for P2; HRV
1
and TLI
2
as inputs.
BIOSIGNALS 2008 - International Conference on Bio-inspired Systems and Signal Processing
322
Table 1: The Mamdani-type fuzzy rules after optimization
and their corresponding weights for P2 with the inputs
HRV
1
and TLI
2
.
No
Rule
1
If HRV1 is M and TLI2 is S then TIR is VH (0.197)
2
If HRV1 is M and TLI2 is S then TIR is VH (0.446)
3
If HRV1 is M and TLI2 is M then TIR is H (0.159)
4 If HRV1 is B and TLI2 is S then TIR is VH (0.527)
5 If HRV1 is M and TLI2 is B then TIR is VH (0.798)
6 If HRV1 is B and TLI2 is M then TIR is H (0.983)
7 If HRV1 is M and TLI2 is B then TIR is H (0.778)
8 If HRV1 is B and TLI2 is B then TIR is N (0.470)
9 If HRV1 is S and TLI2 is B then TIR is L (0.904)
10 If HRV1 is M and TLI2 is VB then TIR is L (0.853)
11
If HRV1 is S and TLI2 is B then TIR is N (0.010)
12
If HRV1 is S and TLI2 is B then TIR is N (0.013)
13
If HRV1 is B and TLI2 is M then TIR is N (0.313)
14 If HRV1 is B and TLI2 is VB then TIR is N (0.864)
15
If HRV1 is B and TLI2 is B then TIR is N (0.331)
16
If HRV1 is VB and TLI2 is M then TIR is N (0.352)
17 If HRV1 is VB and TLI2 is M then TIR is N (0.906)
18 If HRV1 is B and TLI2 is M then TIR is VH (0.819)
Tables 2 and 3 show the model MSE’s and the
correlation factors for the three subjects data which
only justify the initial choice of the criteria proposed
for choosing the candidates' inputs and show that the
model output is improved by using HRV
1
instead of
HRV
2.
0 2 4 6 8 10 12 14 16 18
50
60
70
80
90
100
Tim e in range (% )
(a) Comparison between model output and training data
Mamdani model output
Training data
0 2 4 6 8 10 12 14 16 18
30
40
50
60
70
80
90
100
Time index (T
s
=7.5 min)
Time in range (% )
(b) Comparison between model output and checking data
Mamdani model output
Checking data
Figure 4: Model output of the GA Mamdani-type model of
P2 for TLI
2
and HRV
2
as inputs.
Table 2: Training and testing MSEs and correlations of
Mamdani fuzzy model for P2, P4 and P10 when inputs are
HRV1and TLI2
MSE Correlation
Error
Factor
No
Train Check Train Check
2
inputs
P2 6.7506 130.340 0.983 0.712 2.931
P4 1.0860 93.672 0.997 0.8304 1.022
P10 8.4722 67.533 0.965 0.664 2.578
Table 3: Training and testing MSE and correlation values
of the Mamdani fuzzy model for P2, P4 and P10 when the
inputs are HRV2 andTLI2.
MSE Correlation
Error
Factor
No
Train Check Train Check
2
inputs
P2 7.213 194.930 0.981 0.518 4.383
P4 2.455 478.763 0.986 0.112 5.227
P10 2.840 130.624 0.988 0.541 4.987
4 THE NEW FRAMEWORK FOR
REAL-TIME ADAPTIVE
AUTOMATION
The adaptive fuzzy models developed previously
allow for the OFSs to be used as bio-feedback
signals in order to switch operations between human
and machine. Hence, a conceptual adaptive
automation control system built around aCAMS for
the automation tasks is proposed as shown in Figure
5. The system was implemented using MFC (Visual
C++ 8.0, Microsoft, USA) on a Window-XP
computer. Psycho-physiological signals were
collected using the BioSemi system with the
recording scheme as described in Section 2.2. The
two peripherals, aCAMS and BioSemi computers,
communicate with the host system through Ethernet
networking that uses the TCP/IP communication
protocol.
Figure 6 shows a conceptual automation control
system with the developed fuzzy OFS model for
predictive control and primary task performance for
immediate feedback reaction. The model analyzes
psychophysiological responses every 128 s to
provide information of how the system may drift
into ‘error’. Once a possible system abnormality is
foreseen, the LOA Reallocator either switches
system operation from human to machine or changes
the level of automation (LOA). A “System in Error”
A NEW FRAMEWORK FOR REAL-TIME ADAPTIVE FUZZY MONITORING AND CONTROL FOR HUMANS
UNDER PSYCHOPHYSIOLOGICAL STRESS
323
reported by aCAMS represents an anticipated
system catastrophe if the system operation is not
immediately intervened. The occurrence of such a
fault elicits the LOA Reallocator for immediate
automation intervention. This feedback correction is
synchronized with aCAMS, 1 s in this case. Once an
error occurs, the control is brought to a hysteresis
loop which imposes a refractory duration to LOA
commands to avoid adversary chattering effect.
This coordinating scheme assures function allocation
between human and machine for persistent system
safety and operation performance.
interaction interface
(displays, controls)
Cabin Air
Management
(radio controlled)
in space
aCAMS
on earth
OFS
detector
and
predictor
auto/
manual
fuzzy-logic
control
closed loop
system,
adaptive
automation
interaction interface
(displays, controls)
Cabin Air
Management
(radio controlled)
in space
aCAMS
on earth
OFS
detector
and
predictor
auto/
manual
fuzzy-logic
control
closed loop
system,
adaptive
automation
Figure 5: Conceptual adaptive automation control for the
aCAMS human-machine system.
OFS
HRV1
aCAMS
System
Error
in
Fuzzy Model
Predictor
TLI2
LOA
Reallocator
OR
Reset
120 s
1 s
aCAMS
Simulator
(18 rules)
Fuzzy
Psychophysiology
Process Performance
4 MFs
4 MFs
TIR
Switc hing
Operator
(4MFs)
Figure 6: The control system of adaptive automation with
OFS prediction and process feedback.
Figure 7 demonstrates the screenshot of a
tentative experiment for which only the feedback
correction loop of Figure 6 was activated. The
screenshot shows aCAMS performance,
psychophysiological responses, LOA allocation
commands, subjective ratings, and system
communication status on line. The automation
controller took over the operation task from the
operator and re-allocated LOA immediately
responding to the occurrence of a system
abnormality. The system operation recovered to a
normal state subject to the LOA manipulation.
Figure 7: Screenshot of a tentative system operation. Top-
left: aCAMS performance; top-right: psychophysiological
response; bottom-left: LOA allocation; bottom-right:
subjective ratings; status bar: monitoring of the system
communication.
5 CONCLUSIONS
The first part of this paper related to the elicitation
of ANFIS and Mamdani-type models for identifying
OFSs using psychophysiological and performance
measures. Model analyses revealed that the GA-
based Mamdani-type model generalised better across
the data used and that HRV
1
and TLI
2
represented
the best correlating inputs to the performance output
‘time in range’. The model represents a concise,
transparent (easily understandable) and robust
characterization of OFS and can be easily extended
or modified to accommodate additional input
variables, membership functions and fuzzy rules.
The identification of these OFSs paved the way for
proposing a new framework the real-time
monitoring and adaptive control of automation in
complex and safety-critical human-machine systems.
Preliminary simulation studies using aCAMS, the
OFSs predictor and the LOA fuzzy decision-maker
showed that successful switching of system
automation is possible. It is hoped that real-time
experiments involving the same group of volunteers
who partook in earlier experiments whose data were
used for modelling will be conducted in the near
future.
ACKNOWLEDGEMENTS
All authors wish to acknowledge financial support for this
research work from the UK-EPSRC under Grant
GR/S66985/01. A Nassef wishes to thank his sponsor; the
Egyptian Cultural Bureau in London (UK), for its
BIOSIGNALS 2008 - International Conference on Bio-inspired Systems and Signal Processing
324
financial support and C H Ting gratefully acknowledges
the support of a research leave from The National Chiayi
University, Taiwan.
REFERENCES
Fehrenberg, J., and Wientjes, C. W. J., 2000, ‘Recording
methods in applied environments’ in Engineering
psychology: issues and applications, ed. R. W. Backs
& W. Boucsein W, Erlbaum, Mahawah, pp. 111-136.
Geveins, A. & Smith, M. E., 1999, ‘Detecting transient
cognitive impairment with EEG pattern recognition
methods’ Aviation, Space, and Environmental
Medicine, vol. 70, pp. 1018-1024.
Gevins, A., Smith, E., McEvoy, L., & Yu, D., 1997,
‘High-resolution EEG mapping of cortical activation
related to working memory: Effects of task difficulty
type of processing, and practice’ Cerebral Cortex.,
vol. 7, pp. 374-385.
Goldberg, D. E., 1989, Genetic Algorithms in Search,
Optimization and Machine Learning, Addison-
Wesley.
Hockey, G. R. J., Gaillard, A. W. K. & Burov, O., 2003,
Operator functional State: the assessment and
prediction of human performance degradation in
complex tasks, IOS Press, Amesterdam, The
Netherlands.
Hockey, G. R. L., Wastell, D., & Saucer J., 1998, ‘Effects
of sleep deprivation and user-interface on complex
performance: a multilevel analysis of compensatory
control’ Human Factors, vol. 40, pp. 233-253.
Jang, J., 1993, ‘ANFIS: Adaptive-network-based fuzzy
inference system’ IEEE Transations on Systems, Man
and Cybernetics, vol. 23, pp. 665-685.
Jasper, H. H., 1958, ‘Report of the committee on methods
of clinical examination in electroencephalography’
Electroencephalography and Clinical
Neurophysiology, vol. 10, pp. 370-375.
Kaber, D. B., Riley, J. M., Kheng-Wooi, T. & Endsley, M.
R., 2001, ‘On the design of adaptive automation for
complex systems’ International Journal of Cognitive
Ergonomics, vol. 5, pp. 37-57.
Lorenz, B. and Parasuraman, R., 2003, ‘Human operator
functional state in automated systems: the role of
compensatory control strategies’ In Operator
functional State: the assessment and prediction of
human performance degradation in complex tasks, ed.
G. R. J. Hokey, A. W. K. Gaillard & O. Burov, pp.
224-237, IOS Press, Amesterdam, The Netherlands.
Lorenz, B., 2002, ‘Detection and prediction of an
automation-induced state of impaired operator
competence’ In Proceedings of NATO ARW on
Operator Functional State, Il Ciocco.
Mahfouf, M., Zhang, J., Linkens, D. A. Nassef, A.,
Nickel, P., Hockey, G. R. J., & Roberts, A.C., 2006,
Adaptive Fuzzy Approaches to Modelling Operator
Functional States in a Human-Machine Process
Control System. In Proceedings of FUZZIEEE2007,
London, UK, July 23-26..
Mamdani, E. H, 1974, ‘Applications of fuzzy algorithms
for control of simple dynamic plant’ In Proceedings
IEEE, (121), pp. 1585–1588.
Nickel, P., Roberts, A. C., & Hockey, G. R. J., 2005,
Assessment of high risk operator functional state
markers in dynamical systems – preliminary results
and implications In Proc. of Human Factors and
Ergonomics Society Europe Chapter Annual Meeting
2005, Turin, Italy, Oct. 26-28.
Zadeh, L. A., 1965, ‘Fuzzy sets’ Information and Control,
vol. 8, pp. 338-353.
Zhang, J., Nassef, A., Mahfouf, M., Linkens, D. A., El-
Samahy, E., Hockey, G. R. J., Nickel, P. & Roberts,
A. C., 2006. Modelling and analysis of HRV under
physical and mental workloads. In Proc. of the 6
th
IFAC Symposium on Modelling and Control in
Biomedical Systems, Reims, France, Sept. 20-22, pp.
189-194.
A NEW FRAMEWORK FOR REAL-TIME ADAPTIVE FUZZY MONITORING AND CONTROL FOR HUMANS
UNDER PSYCHOPHYSIOLOGICAL STRESS
325