A NEW FRAMEWORK FOR REAL-TIME ADAPTIVE FUZZY

MONITORING AND CONTROL FOR HUMANS UNDER

PSYCHOPHYSIOLOGICAL STRESS

A. Nassef, C. H. Ting, M. Mahfouf, D. A. Linkens

Department of Automatic Control and Systems, The University of Sheffield, Sheffield, United Kingdom

P. Nickel, G. R. J. Hockey, A. C. Roberts

Department of Psychology, The University of Sheffield, Sheffield, United Kingdom

Keywords: Adaptive Automation, Operator Functional State, Cardiovascular System, Electroencepharograph, Fuzzy

Systems, Genetic Algorithms, Signal Processing.

Abstract: This paper proposes a new framework for the on-line monitoring and adaptive control of

psychophysiological markers relating to humans under stress. The starting point of this framework relates to

the assessment of the so-called operator functional state (OFS) using physical as well as psychological

measures. An adaptive neural-fuzzy model linking Heart-Rate Variability (HRV) and Task Load Index

(TLI) with the subjects’ optimal performance has been elicited and validated via a series of real-life

experiments involving process control tasks simulated on an Automation-Enhanced Cabin Air Management

System (aCAMS). The elicited model has been used as the basis for an on-line control system, whereby the

model predictions which indicate whether the actual system is in error or not, have been used to modify the

level of automation which the system may operates under.

1 INTRODUCTION

With increasingly complex design of automation in

safety-critical applications, there is a growing

concern for the consequences of performance

breakdown. This is because the human operator’s

role has become compromised with increasing

operational demand, stress and fatigue, which all

threaten safety and reliability (Hockey et al., 2003).

The approach taken to this problem in this paper is

based on an ‘Operator Functional State’ (OFS)

framework in which the performance of the operator

is constrained by requirements to manage the

automation tasks and his/her own personal state.

The OFS model should predict that, for a period

before manifest breakdown occurs, the operator will

be in a vulnerable state, because of reduced spare

capacity to respond to emergencies. The goal of the

current programme of work is to develop models for

evaluating psychophysiological markers of this high

risk strain state. If such states can be reliably

detected, they can be used to trigger a switch of

control from human to computer, through an

adaptive automation (AA) interface, reducing the

risk of operational breakdown (Kaber et al. 2001).

A likely marker is the ‘task load index’ (TLI)

identified by Gevins and his group (Gevins and

Smith, 1999). TLI is based on the presence of high

levels of theta activity at frontal midline sites, with

concomitant attenuation of alpha power in parietal

sites [theta/alpha]. Observation of reduced frontal-

midline theta power may reflect direct effects of

fatigue or strategic disengagement from the

executive requirements of the task management

(Lorenz and Parasuraman, 2003).

To investigate this, a task known as automation-

enhanced Cabin Air Management System (aCAMS)

(Figure 1), developed by Hockey and colleagues

(Hockey et al., 1998, Lorenz, 2002) to simulate the

atmospheric environment within a space capsule, is

used. This semi-automatic system required operators

to maintain an appropriate quantity and quality of

breathable air by keeping system parameters

(temperature, humidity, pressure, O

, CO

) within

320

Nassef A., H. Ting C., Mahfouf M., A. Linkens D., Nickel P., R. J. Hockey G. and C. Roberts A. (2008).

A NEW FRAMEWORK FOR REAL-TIME ADAPTIVE FUZZY MONITORING AND CONTROL FOR HUMANS UNDER PSYCHOPHYSIOLOGICAL

STRESS.

In Proceedings of the First International Conference on Bio-inspired Systems and Signal Processing, pages 320-325

DOI: 10.5220/0001066003200325

 SciTePress

normal ranges (primary task). The operators

interacted with a dynamic visual display that

provides data on system variables and functions via

a range of controls and automation tools; this is a

large mental burden to the operator.

scrubber

Heater

Cooler

CABIN

Vent

Dehumidifier

Oxygen

Nitrogen

Temperature

Pressure

Humidity

Tasks

Technology

Humans

scrubber

Heater

Cooler

CABIN

Vent

Dehumidifier

Oxygen

Nitrogen

Temperature

Pressure

Humidity

Tasks

Technology

Humans

Figure 1: The aCAMS human-machine system.

The main objective of the research work presented

in this paper is to propose a new framework for the

on-line (real-time) monitoring of the human

operator’s performance for breakdown, stress or

fatigure and the adaptive control of the level of

automation. In order to achieve this a model that

describes the input and output relationship between

the psychophysiological measures (e.g.

cardiovascular and EEG activities) and functional

(i.e. cognitive, mental or psychological) states of the

operator in a simulated process control environment

is built first. The model can then be implemented in

an adaptive automation control system to represent a

kernel in OFS estimation. In the present

investigation, the OFSs identification is achieved by

using adaptive fuzzy modelling which requires the

measured psychophysiological and primary task

performance data only. The proposed modelling

approaches are shown by simulation results to be

capable of effectively exploiting the information

contained in the measured physiological and

performance data. By using this model the OFS may

be identified or predicted by monitoring the changes

in the psychophysiological and performance data,

and hence the model output can be used as a bio-

feedback signal in closed-loop automation control.

This paper is organised as follows: Section 2 will

outline the chosen technical paradigm behind the

intelligent systems-based modelling strategy.

Section 3 will present the final models which were

adopted and Section 4 shows how such models can

be included in the real-time framework for

monitoring and adaptive control. Finally, Section 5

will draw some conclusions in relation to this overall

research study.

2 FUZZY MODELLING OF

OPERATOR FUNCTIONAL

STATE (OFS)

For the purpose of modelling fuzzy logic (Zadeh,

1965) was chosen as the main paradigm for

characterising the input/output mappings because of

its tolerance to uncertainties and also for the fact it

can model human perception in a transparent way

without a greater loss in accuracy. As a result, two

types of fuzzy models were constructed and

optimised automatically: one using neural networks

leading to the Artificial Network Fuzzy Inference

System (ANFIS) architecture (Jang, 1993) which

utilises and the other using Genetic Algorithms

(Goldberg, 1989) to estimate the parameters of the

membership functions and the fuzzy rules of a

Mamdani-type structure (Mamdani, 1974). In order

to carry-out this modelling operation successfully it

is important to first specify the variables associated

with this input/output mapping and then carry-out

the real-time experiments (Mahfouf et al., 2006)

which will enable one to collect the input/output

data information as will be explained next.

2.1 Model Inputs and Output

The candidate inputs of the fuzzy model may

include Heart Rate Variability (HRV) and EEG

markers (TLI), which were found to be most

sensitive to the changes in mental workload

((Fehrengerg and Wientjes, 2000);Nickel et al.,

2005; Zhang et al., 2006). The optimal number of

inputs selected from the above candidate inputs was

determined by linear correlation analysis of the

relationship between the input and output data. The

single output of the model is ‘Time in Range’ related

to the primary task performance.

2.2 Data Acquisition and Analysis

The BioSemi® system (Biosemi, the Netherland)

was used for EEG recording at 32 electrode sites

defined by the international 10-20 system (Jasper,

1958). The electrodes were re-referenced to two

linked mastoids. The EEG signal, sampled at a rate

of 2048 Hz, was pre-processed with a band-pass

filter between 1.6 and 25 Hz. The power in the three

bands (i.e., theta, alpha and beta) for each of the

selected electrode sites was calculated. The primary-

task performance data (‘Time in Range’) were

sampled every 1 min.

The heart rate (HR) signal was recorded every 1

s as soon as the aCAMS was started up. HRV

defined as the average of the 0.1 Hz component

A NEW FRAMEWORK FOR REAL-TIME ADAPTIVE FUZZY MONITORING AND CONTROL FOR HUMANS

UNDER PSYCHOPHYSIOLOGICAL STRESS

321

powers. HRV

is defined as the HR variation

coefficient and given by the following expression:

HRV

(1)

where σ and μ denote the standard deviation and

average of a HR segment of 7.5 min.

The TLI calculated using different EEG band

powers was proposed in (Gevins et al., 1997). The

TLI indices, TLI

and TLI

used in this paper, are

given as follows:

⎪

⎩

⎪

⎨

⎧

POCP

TLI

(2)

where

P and

P denote the theta- and alpha-band

power, respectively; the EEG frequency bands are

defined in order as: θ, Fz: 6-7 Hz; α, Pz: 10-12 Hz;

θ, AFz: 5-7 Hz; α, CPz: 8-10.5 Hz; α, POz: 10-13.5

Hz; and Fz, Pz, AFz, CPz, and POz are the five EEG

electrode sites on the scalp introduced in the

standard 10-20 system (Jasper, 1958).

3 RESULTS AND DISCUSSIONS

In this simulation the signal data sampling interval

was taken to be 7.5 min and Gaussian MFs were

used for both fuzzy models. The choice of the

candidate input was mainly driven by the value of

the input-output correlation factor (the higher the

better), the training and testing data correlation

factor (the higher the better) and the MSE values of

the training and testing data. As a result, the two

inputs HRV

and TLI

were selected for both fuzzy

models. The training and testing data set was

obtained from the 1

and 2

experimental sessions,

respectively. The ANFIS modelling result for P2 is

shown in Figure 2.

Due to the large differences between the MSE

values of the model output for each subject another

index was introduced to differentiate between

models. This index was named "Error Factor" and is

defined by the ratio between the MSE of the model

output when using the validating data and the MSE

between the training and validating data as shown in

Equ. (3).

chk-Tr

chk-output model

FactorError

MSE

(3)

Using this new index it was found that Subjects

P2, P4, and P10 led to the highest values, i.e. the

worst performing models compared to the other

subjects. So, those subjects' data have been chosen

for the next study. The optimised rules of Mamdani-

type fuzzy model and their weights are illustrated in

Table 1. The optimal MFs and degrees of belief

(rules’ weight) in each rule are identified by using a

GA approach. It is noted that the 1

, 2

, 3

, 11

, 13

, 15

and 16

rules (see Table 1 in ‘bold’

characters) are less important in terms of the smaller

weights. The comparison of the model output and

desired output is shown in Figure 3 for P2. Figure 4

illustrates the model output when HRV2 and TLI2

are used as inputs

0 2 4 6 8 10 12 14 16 1

100

Time in range (%)

(a) Comparison of ANFIS output and training data (P2)

Training data

Model output

0 2 4 6 8 10 12 14 16 1

100

Time index (T

= 7.5 min)

Time in range (%)

(b) Comparison of ANFIS output and checking data (P2)

Checking data

Model output

Figure 2: ANFIS modelling results for P2; HRV

and TLI

as inputs.

0 2 4 6 8 10 12 14 16 18

100

Time in range (%)

(a) Comparison between model output and training data

Mamdani model output

Training data

0 2 4 6 8 10 12 14 16 18

100

Time index (T

=7.5 min)

Time in range (%)

(b) Comparison between model output and checking data

Mamdani model output

Checking data

Figure 3: Modelling results via the GA-based Mamdani-

type model for P2; HRV

and TLI

as inputs.

BIOSIGNALS 2008 - International Conference on Bio-inspired Systems and Signal Processing

322

Table 1: The Mamdani-type fuzzy rules after optimization

and their corresponding weights for P2 with the inputs

HRV

and TLI

Rule

If HRV1 is M and TLI2 is S then TIR is VH (0.197)

If HRV1 is M and TLI2 is S then TIR is VH (0.446)

If HRV1 is M and TLI2 is M then TIR is H (0.159)

4 If HRV1 is B and TLI2 is S then TIR is VH (0.527)

5 If HRV1 is M and TLI2 is B then TIR is VH (0.798)

6 If HRV1 is B and TLI2 is M then TIR is H (0.983)

7 If HRV1 is M and TLI2 is B then TIR is H (0.778)

8 If HRV1 is B and TLI2 is B then TIR is N (0.470)

9 If HRV1 is S and TLI2 is B then TIR is L (0.904)

10 If HRV1 is M and TLI2 is VB then TIR is L (0.853)

If HRV1 is S and TLI2 is B then TIR is N (0.010)

If HRV1 is S and TLI2 is B then TIR is N (0.013)

If HRV1 is B and TLI2 is M then TIR is N (0.313)

14 If HRV1 is B and TLI2 is VB then TIR is N (0.864)

If HRV1 is B and TLI2 is B then TIR is N (0.331)

If HRV1 is VB and TLI2 is M then TIR is N (0.352)

17 If HRV1 is VB and TLI2 is M then TIR is N (0.906)

18 If HRV1 is B and TLI2 is M then TIR is VH (0.819)

Tables 2 and 3 show the model MSE’s and the

correlation factors for the three subjects data which

only justify the initial choice of the criteria proposed

for choosing the candidates' inputs and show that the

model output is improved by using HRV

instead of

HRV

0 2 4 6 8 10 12 14 16 18

100

Tim e in range (% )

(a) Comparison between model output and training data

Mamdani model output

Training data

0 2 4 6 8 10 12 14 16 18

100

Time index (T

=7.5 min)

Time in range (% )

(b) Comparison between model output and checking data

Mamdani model output

Checking data

Figure 4: Model output of the GA Mamdani-type model of

P2 for TLI

and HRV

as inputs.

Table 2: Training and testing MSEs and correlations of

Mamdani fuzzy model for P2, P4 and P10 when inputs are

HRV1and TLI2

MSE Correlation

Error

Factor

Train Check Train Check

inputs

P2 6.7506 130.340 0.983 0.712 2.931

P4 1.0860 93.672 0.997 0.8304 1.022

P10 8.4722 67.533 0.965 0.664 2.578

Table 3: Training and testing MSE and correlation values

of the Mamdani fuzzy model for P2, P4 and P10 when the

inputs are HRV2 andTLI2.

MSE Correlation

Error

Factor

Train Check Train Check

inputs

P2 7.213 194.930 0.981 0.518 4.383

P4 2.455 478.763 0.986 0.112 5.227

P10 2.840 130.624 0.988 0.541 4.987

4 THE NEW FRAMEWORK FOR

REAL-TIME ADAPTIVE

AUTOMATION

The adaptive fuzzy models developed previously

allow for the OFSs to be used as bio-feedback

signals in order to switch operations between human

and machine. Hence, a conceptual adaptive

automation control system built around aCAMS for

the automation tasks is proposed as shown in Figure

5. The system was implemented using MFC (Visual

C++ 8.0, Microsoft, USA) on a Window-XP

computer. Psycho-physiological signals were

collected using the BioSemi system with the

recording scheme as described in Section 2.2. The

two peripherals, aCAMS and BioSemi computers,

communicate with the host system through Ethernet

networking that uses the TCP/IP communication

protocol.

Figure 6 shows a conceptual automation control

system with the developed fuzzy OFS model for

predictive control and primary task performance for

immediate feedback reaction. The model analyzes

psychophysiological responses every 128 s to

provide information of how the system may drift

into ‘error’. Once a possible system abnormality is

foreseen, the LOA Reallocator either switches

system operation from human to machine or changes

the level of automation (LOA). A “System in Error”

A NEW FRAMEWORK FOR REAL-TIME ADAPTIVE FUZZY MONITORING AND CONTROL FOR HUMANS

UNDER PSYCHOPHYSIOLOGICAL STRESS

323

reported by aCAMS represents an anticipated

system catastrophe if the system operation is not

immediately intervened. The occurrence of such a

fault elicits the LOA Reallocator for immediate

automation intervention. This feedback correction is

synchronized with aCAMS, 1 s in this case. Once an

error occurs, the control is brought to a hysteresis

loop which imposes a refractory duration to LOA

commands to avoid adversary chattering effect.

This coordinating scheme assures function allocation

between human and machine for persistent system

safety and operation performance.

interaction interface

(displays, controls)

Cabin Air

Management

(radio controlled)

in space

aCAMS

on earth

OFS

detector

and

predictor

auto/

manual

fuzzy-logic

control

closed loop

system,

adaptive

automation

interaction interface

(displays, controls)

Cabin Air

Management

(radio controlled)

in space

aCAMS

on earth

OFS

detector

and

predictor

auto/

manual

fuzzy-logic

control

closed loop

system,

adaptive

automation

Figure 5: Conceptual adaptive automation control for the

aCAMS human-machine system.

OFS

HRV1

aCAMS

System

Error

Fuzzy Model

Predictor

TLI2

LOA

Reallocator

Reset

120 s

1 s

aCAMS

Simulator

(18 rules)

Fuzzy

Psychophysiology

Process Performance

4 MFs

TIR

Switc hing

Operator

(4MFs)

Figure 6: The control system of adaptive automation with

OFS prediction and process feedback.

Figure 7 demonstrates the screenshot of a

tentative experiment for which only the feedback

correction loop of Figure 6 was activated. The

screenshot shows aCAMS performance,

psychophysiological responses, LOA allocation

commands, subjective ratings, and system

communication status on line. The automation

controller took over the operation task from the

operator and re-allocated LOA immediately

responding to the occurrence of a system

abnormality. The system operation recovered to a

normal state subject to the LOA manipulation.

Figure 7: Screenshot of a tentative system operation. Top-

left: aCAMS performance; top-right: psychophysiological

response; bottom-left: LOA allocation; bottom-right:

subjective ratings; status bar: monitoring of the system

communication.

5 CONCLUSIONS

The first part of this paper related to the elicitation

of ANFIS and Mamdani-type models for identifying

OFSs using psychophysiological and performance

measures. Model analyses revealed that the GA-

based Mamdani-type model generalised better across

the data used and that HRV

and TLI

represented

the best correlating inputs to the performance output

‘time in range’. The model represents a concise,

transparent (easily understandable) and robust

characterization of OFS and can be easily extended

or modified to accommodate additional input

variables, membership functions and fuzzy rules.

The identification of these OFSs paved the way for

proposing a new framework the real-time

monitoring and adaptive control of automation in

complex and safety-critical human-machine systems.

Preliminary simulation studies using aCAMS, the

OFSs predictor and the LOA fuzzy decision-maker

showed that successful switching of system

automation is possible. It is hoped that real-time

experiments involving the same group of volunteers

who partook in earlier experiments whose data were

used for modelling will be conducted in the near

future.

ACKNOWLEDGEMENTS

All authors wish to acknowledge financial support for this

research work from the UK-EPSRC under Grant

GR/S66985/01. A Nassef wishes to thank his sponsor; the

Egyptian Cultural Bureau in London (UK), for its

BIOSIGNALS 2008 - International Conference on Bio-inspired Systems and Signal Processing

324

financial support and C H Ting gratefully acknowledges

the support of a research leave from The National Chiayi

University, Taiwan.

REFERENCES

Fehrenberg, J., and Wientjes, C. W. J., 2000, ‘Recording

methods in applied environments’ in Engineering

psychology: issues and applications, ed. R. W. Backs

& W. Boucsein W, Erlbaum, Mahawah, pp. 111-136.

Geveins, A. & Smith, M. E., 1999, ‘Detecting transient

cognitive impairment with EEG pattern recognition

methods’ Aviation, Space, and Environmental

Medicine, vol. 70, pp. 1018-1024.

Gevins, A., Smith, E., McEvoy, L., & Yu, D., 1997,

‘High-resolution EEG mapping of cortical activation

related to working memory: Effects of task difficulty

type of processing, and practice’ Cerebral Cortex.,

vol. 7, pp. 374-385.

Goldberg, D. E., 1989, Genetic Algorithms in Search,

Optimization and Machine Learning, Addison-

Wesley.

Hockey, G. R. J., Gaillard, A. W. K. & Burov, O., 2003,

Operator functional State: the assessment and

prediction of human performance degradation in

complex tasks, IOS Press, Amesterdam, The

Netherlands.

Hockey, G. R. L., Wastell, D., & Saucer J., 1998, ‘Effects

of sleep deprivation and user-interface on complex

performance: a multilevel analysis of compensatory

control’ Human Factors, vol. 40, pp. 233-253.

Jang, J., 1993, ‘ANFIS: Adaptive-network-based fuzzy

inference system’ IEEE Transations on Systems, Man

and Cybernetics, vol. 23, pp. 665-685.

Jasper, H. H., 1958, ‘Report of the committee on methods

of clinical examination in electroencephalography’

Electroencephalography and Clinical

Neurophysiology, vol. 10, pp. 370-375.

Kaber, D. B., Riley, J. M., Kheng-Wooi, T. & Endsley, M.

R., 2001, ‘On the design of adaptive automation for

complex systems’ International Journal of Cognitive

Ergonomics, vol. 5, pp. 37-57.

Lorenz, B. and Parasuraman, R., 2003, ‘Human operator

functional state in automated systems: the role of

compensatory control strategies’ In Operator

functional State: the assessment and prediction of

human performance degradation in complex tasks, ed.

G. R. J. Hokey, A. W. K. Gaillard & O. Burov, pp.

224-237, IOS Press, Amesterdam, The Netherlands.

Lorenz, B., 2002, ‘Detection and prediction of an

automation-induced state of impaired operator

competence’ In Proceedings of NATO ARW on

Operator Functional State, Il Ciocco.

Mahfouf, M., Zhang, J., Linkens, D. A. Nassef, A.,

Nickel, P., Hockey, G. R. J., & Roberts, A.C., 2006,

Adaptive Fuzzy Approaches to Modelling Operator

Functional States in a Human-Machine Process

Control System. In Proceedings of FUZZIEEE2007,

London, UK, July 23-26..

Mamdani, E. H, 1974, ‘Applications of fuzzy algorithms

for control of simple dynamic plant’ In Proceedings

IEEE, (121), pp. 1585–1588.

Nickel, P., Roberts, A. C., & Hockey, G. R. J., 2005,

Assessment of high risk operator functional state

markers in dynamical systems – preliminary results

and implications In Proc. of Human Factors and

Ergonomics Society Europe Chapter Annual Meeting

2005, Turin, Italy, Oct. 26-28.

Zadeh, L. A., 1965, ‘Fuzzy sets’ Information and Control,

vol. 8, pp. 338-353.

Zhang, J., Nassef, A., Mahfouf, M., Linkens, D. A., El-

Samahy, E., Hockey, G. R. J., Nickel, P. & Roberts,

A. C., 2006. Modelling and analysis of HRV under

physical and mental workloads. In Proc. of the 6

IFAC Symposium on Modelling and Control in

Biomedical Systems, Reims, France, Sept. 20-22, pp.

189-194.

A NEW FRAMEWORK FOR REAL-TIME ADAPTIVE FUZZY MONITORING AND CONTROL FOR HUMANS

UNDER PSYCHOPHYSIOLOGICAL STRESS

325