Egon L. van den Broek
Center for Telematics and Information Technology (CTIT), University of Twente
P.O. Box 217, 7500 AE Enschede, The Netherlands
Joris H. Janssen
, Joyce H. D. M. Westerink
User Experience Group, Philips Research Europe, High Tech Campus 34, 5656 AE Eindhoven, The Netherlands
Jennifer A. Healey
Intel Corporation, Corporate Technology Group, 3600 Juliette Lane SC12-319 Santa Clara CA 95054, U.S.A.
Affective signal processing (ASP), Emotion, Validation, Physiology-driven, Triangulation.
Although emotions are embraced by science, their recognition has not reached a satisfying level. Through a
concise overview of affect, its signals, features, and classification methods, we provide understanding for the
problems encountered. Next, we identify the prerequisites for successful Affective Signal Processing: valida-
tion (e.g., mapping of constructs on signals), triangulation, a physiology-driven approach, and contributions
of the signal processing community. Using these directives, a critical analysis of a real-world case is provided.
This illustrates that the prerequisites can become a valuable guide for Affective Signal Processing.
When dealing with people, let us remember that we
are not dealing with creatures of logic; we are dealing
with creatures of emotion ... (Dale Carnegie, 1936)
Dale Carnegie (1888-1955)
While a century ago emotions were considered as too
spiritual and human’s health was solely explained in
physical (e.g., injuries) and physiological terms (e.g.,
bacteria, viruses), it is now generally acknowledged
that emotions have their impact on health and illness.
It has been shown that emotions influence our cog-
nitive and social functioning as well as our cardio-
vascular system (Schuler and O’Brien, 1997) and, as
a consequence, can even either shorten or prolong
Joris H. Janssen and Egon L. van den Broek did equally
contribute to this article; hence, they are shared first authors.
For the interested reader, we refer to “Historical foun-
dations of social effectiveness? Dale Carnegie’s princi-
ples” (Duke and Novicevic, 2008), which illustrates the
timeless significance of Carnegie’s work.
life (Frederickson et al., 2000).
Medicine’s interest in emotions was followed by
that of Artificial Intelligence (AI), which envisioned
that emotions both lead the path to true AI and en-
hance the communication between man and machine
(or even environment) (Picard, 1997; Minsky, 2006).
This expresses the intrinsic need for automated sens-
ing of emotions. Often this is done through speech
or face analysis; see Cowie et al. (2001) and Zeng
et al. (in press) for reviews. Alternatively, physio-
logical signals are used to identify emotions; see also
Table 1 and 2, Box 1, and (Westerink et al., 2008a).
This paper discusses the this last approach, which we
will denote as Affective Signal Processing (ASP).
Physiological signals have the advantage that they
are free from social masking and have the potential of
being measured by non-invasive sensors; e.g., (Sam-
boa et al., 2008; Westerink et al., 2008a), making
them suited for a wide range of applications. In
contrast, recognizing facial expressions is notoriously
problematic and speech is often either absent or suf-
fers from severe distortions in many in real-world ap-
plications (Cowie et al., 2001; Zeng et al., in press).
In the next section, we provide both an overview
L. van den Broek E., H. Janssen J., H. D. M. Westerink J. and A. Healey J. (2009).
In Proceedings of the International Conference on Bio-inspired Systems and Signal Processing , pages 426-433
DOI: 10.5220/0001780504260433
and a review of ASP. After that, we introduce four
prerequisites for successful ASP. Moreover, a real
world case will be analyzed, using these prerequisites
(Box 1). We finish the paper with a brief conclusion
in which we denote some applications of ASP.
A broad range of affective signals are used in affec-
tive sciences. When processing such signals some
general issues have to be taken in consideration: 1)
Affective signals are typically derived through non-
invasive methods to determine changes in physiology
and, as such, are indirect measures. Hence, a delay
between the actual change in emotional state and the
recorded change in signal has to be taken into ac-
count. 2) Physiological sensors are unreliable: they
contain movement artifacts and are sensitive to dif-
ferences in bodily position. 3) Most sensors are ob-
trusive, preventing their integration in real world ap-
plications. 4) Affective signals are influenced by (the
interaction among) a variety of factors (Cacioppo and
Tassinary, 1990). Some of these sources are located
internally (e.g., a thought) and some are among the
broad range of possible external factors (e.g., a sig-
nal outside). This makes affective signals inherently
noisy, which is most prominent in real world research.
5) Physiological changes can evolve in a matter of
milliseconds, seconds, minutes or even longer. Some
changes hold for only a brief moment, while others
can even be permanent. Although seldom reported,
the expected time windows of change are of interest.
In particular since changes can add to each other, even
when having a different origin. 6) Humans are no lin-
ear time (translationor shift) invariant systems (Bouc-
sein, 1992), they habituate. This increases the com-
plexity of ASP substantially, since most signal pro-
cessing techniques rely on this assumption. 7) Af-
fective signals have large individual differences. This
calls for methods and models tailored to the indi-
vidual. It has been shown that personal approaches
increase the performance of ASP (Bailenson et al.,
2.1 Classification of Affective Signals
To enable processing of the signals, in most cases
comprehensive sets of features have to be identified
for each affective signal. To extract these features, the
affective signals are processed in the time (e.g., statis-
tical moments (Westerink et al., 2008b)), frequency
(e.g., Fourier), time-frequency (e.g., Wavelets), or
power domain (e.g., Periodogram and Autoregres-
sion). In Table 1, we provide a brief overview of the
signals most often applied, including their best known
features, with reference to their physiological source.
The features obtained from the affective signals
(see Table 1) are fed to pattern recognition methods,
which can be classified as: template matching, syn-
tactic or structural matching, and statistical classifica-
tion; e.g., artificial neural networks (ANN). The for-
mer two are not or seldom used in affective signal pro-
cessing; most ASP schemes use the latter. Statistical
pattern recognition distinguishes supervised and un-
supervised (e.g., clustering) pattern recognition; i.e.,
respectively, with or without a set of (labeled) training
data. With unsupervised pattern recognition, the dis-
tance/similarity measure used and the algorithm ap-
plied to generate the clusters are key elements. Super-
vised pattern recognition relies on learning from a set
of examples (training set). Statistical pattern recog-
nition uses input features, a discriminant function (or
network function for ANN) to recognize the features,
and an error criterion in its classification process.
In the field of ASP, several studies have been con-
ducted, using a broad range of signals, features, and
classifiers; see Table 2 for an overview. Nonetheless,
both the recognition performance and the number of
emotions that the classifiers were able to discriminate
are disappointing. Moreover, comparing the different
studies is problematic because of the different settings
the research was applied in, ranging from controlled
lab studies to real world testing, the type of emotion
triggers used, the number of target states to be dis-
criminated, and the signals and features employed.
This illustrates the need for a set of prerequisites for
3.1 Validity
In the pursuit to trigger emotions in a more or less
controlled manner, a range of methods have been
applied: actors, images (IAPS), sounds (e.g., mu-
sic), (fragments of) movies (Westerink et al., 2008b),
speech (Van den Broek, 2004), commercials (Ha-
zlett and Hazlett, 1999; Poels and Dewitte, 2006),
games, agents / serious gaming / virtual reality (Slater
et al., 2006; Westerink et al., 2008a), reliving of emo-
tions (Van den Broek, 2004), and real world expe-
riences (Healey and Picard, 2005); see also Box 1.
However, how to know which of these methods ac-
tually triggered participants’ true emotions? This is
a typical concern of validity, which is a crucial issue
Table 1: An overview of common physiological signals and features used in ASP.
Physiology Features Unit Remark
Cardiovascular activity Heart rate (HR) beats / min
through ECG or BVP SD IBIs s HRV index
(Berntson et al., 1997) RMSSD IBIs s HRV index
LF power (0.05Hz - 0.15Hz) ms
Sympathetic activity
HF power (0.15HZ - 0.40Hz) ms
Parasympathetic activity
VLF power ( < 0.05Hz) ms
Pulse Transit Time (PTT) ms
Electrodermal Activity (EDA) Mean, SD SCL µS Tonic Sympathetic Activity
(Boucsein, 1992) Nr of SCRs Rate Phasic Activity
SCR amplitude µS Phasic Activity
SCR 1/2 recovery time s
SCR rise time s
Skin temperature (ST) Mean, SD temp
Respiration Respiration rate
(Grossman and Taylor, 2007) Amplitude Resps
Muscle activity Mean, SD corrugator supercilii µV Frowning
through EMG Mean, SD zygomaticus major µV Smiling
(Reaz et al., 2006) Mean, SD upper trapezius µV
(Westerink et al., 2008b) Mean, SD inter-blink interval ms
Notes. SD: Standard deviation; RMSSD: Root Mean Sum of Square Differences; IBI: Inter-beat interval; LF: Low frequency;
HF: High frequency; VLF: Very Low Frequency; SCL: Skin Conductance Level; SCR: Skin Conductance Response; ECG:
Electrocardiogram; EMG: Electromyogram; BVP: Blood volume pulse.
for ASP. Validity can be best obtained through four
approaches: content, criteria-related, construct, and
ecological validation, which we will discuss in rela-
tion to ASP; in addition, see Box 1.
Content validity refers to a) The agreement of ex-
perts on the domain of interest; e.g., limited to a spe-
cific application or group of patients; b) The degree
to which a feature (or its parameters) of a given signal
represents a construct; and c) The degree to which a
set of features (or their parameters) of a given set of
signals adequately represents all facets of the domain.
For instance, employing only skin conductance level
(SCL) for ASP will lead to a weak content validity
when trying to measure emotion, as SCL is known to
relate to the arousal component of an emotion, but not
to the valence component. However, when trying to
measure only emotional arousal, measuring only SCL
may form strong content validity; see also Box 1.
Criteria-related validity handles the quality of the
translation from the preferred measurement to an al-
ternative, rather than to what extend the measure-
ment represents a construct. Emotions are prefer-
ably measured at the moment they occur; however,
measurements before (predictive) or after (postdic-
tive) the particular event are sometimes more feasible;
e.g., through subjectivequestionnaires. The quality of
these translations are referred to as predictive or post-
dictive validity. A third form of criteria-related valid-
ity is concurrent validity: a metric for the reliability
of measurements applied in relation to the preferred
standard. For instance, the more affective states are
discriminated the higher the concurrent validity.
A construct validation process aims to develop a
nomological network (i.e., a ground truth), or possi-
bly an ontology or semantic network, build around
the construct of interest. Such a network requires
theoretically grounded, observable, operational def-
initions of all constructs and the relations between
them. Such a network aims to provide a verifiable
theoretical framework. The lack of such a network
is one of the most pregnant problems ASP is coping
with; e.g., see Box 1. A frequently occurring mis-
take is that emotions are denoted, where moods (i.e.,
longer object-unrelated affective states with very dif-
ferent physiology) are meant. This is very relevant
for ASP, as it is known that moods are accompanied
by very different physiological patterns than emotions
are (Gendolla and Brinkman, 2005).
Ecological validity refers to the influence of the
context on measurements. We identify two issues:
1) Natural affective events are sparse, which makes it
hard to let participants cycle through a range of affec-
BIOSIGNALS 2009 - International Conference on Bio-inspired Systems and Signal Processing
Table 2: A summary of 18 studies that have tried to infer affect from physiological signals.
Information source Signals Part Fea Sel / Red Classifiers Target Result
(Sinha and Parsons, 1996) M 27 18 LDA 2 emotions 86%
(Picard et al., 2001) C ,E ,R ,M 1 40 SFS,
LDA 8 emotions 81%
(Scheirer et al., 2002) C ,E 24 5 Viterbi HMM 2 frustrations 64%
(Nasoz et al., 2003) C ,E ,S 31 3 kNN, LDA 6 emotions 69%
(Takahashi, 2003) C ,E ,B 12 18 SVM 6 emotions 42%
(Haag et al., 2004) C ,E ,S ,M ,R 1 13 MLP val / aro 90%
(Kim et al., 2004) C ,E ,S 175 SVM 3 emotions 78%
(Lisetti and Nasoz, 2004) C ,E ,S 29 kNN, LDA,
6 emotions 84%
(Wagner et al., 2005) C ,E ,R ,M 1 32 SFS,
4 emotions 92%
(Yoo et al., 2005) C ,E 6 5 MLP 4 emotions 80%
(Choi and Woo, 2005) G 3 PCA MLP 4 emotions 90%
(Healey and Picard, 2005) C ,G ,R ,M 9 22 Fisher LDA 3 stress levels 97%
(Rani et al., 2006) C ,G ,S ,M ,P 15 46 kNN, SVM,
3 emotions 85%
(Zhai and Barreto, 2006) C ,G ,S ,P 32 11 SVM 2 stress levels 90%
(Leon et al., 2007) C ,E 8 5 DBI AANN 3 emotions 71%
(Liu et al., 2008) C ,E ,S ,M 6 35 SVM 3 affect states 83%
(Katsis et al., 2008) C ,E ,M ,R 10 15 SVM, ANFIS 4 affect states 79%
(Yannakakis and Hallam,
C ,E 72 20 ANOVA SVM, MLP 2 fun levels 70%
(Kim and Andr´e, in press) C ,E ,M ,R 3 110 SBS LDA 2 fun levels 70%
Notes. Part: the number of participants; Fea: the number of features; Sel / Red: Algorithms used for selection or reduction of
features; C : Cardiovascular activity; E : Electrodermal activity; R : Respiration; M : Electromyogram; B : Electroencephalo-
gram; S : Skin temperature; P : Pupil Diameter; MLP: MultiLayer Perceptron; HMM: Hidden Markov Model; RT: Regression
Tree; BN: Bayesian Network; AANN: Auto-Associative Neural Network; SVM: Support Vector Machine; LDA: Linear
Discriminant Analysis; kNN: k Nearest Neighbors; ANFIS: Adaptive Neuro-Fuzzy Inference System; DBI: Davies-Bouldin
Index; PCA: Principal Component Analysis; SFS: Sequential Forward Selection; SBS: Sequential Backward Selection.
tive states in a limited time frame and 2) The affective
signals that occur are easily contaminated by contex-
tual factors; so, using a similar context as the intended
ASP application for initial learning is of vital impor-
tance; see also Box 1. Although understandable from
a measurement-feasibility perspective, emotion mea-
surements are often done in controlled laboratory set-
tings. This makes results poorly generalizable to real-
world applications. Hence, the need for unobtrusive
sensors and methods to make longitudinal real-world
studies possible is pressing.
3.2 Triangulation
We propose to adopt the principle of Triangulation
on ASP, as applied in social sciences and human-
computer interaction. Heath (2001) defines triangula-
tion as “the strategy of using multiple operationaliza-
tions of constructs to help separate the construct un-
der consideration from other irrelevancies in the op-
erationalization”. Using this strategy provides several
advantages: 1) Distinct signals can be used to vali-
date each other; 2) Extrapolations can be made based
on multiple data sets, providing more certainty. In
turn, corrections can be made to errors in a result set
that clearly defy from other results; and 3) More solid
ground, or even a ground truth (see also Section 3.1),
is obtained for the interpretation of signals, as multi-
ple perspectives are used.
Triangulation was, for example, successfully em-
ployed by Bailenson et al. (2008) and Healey and
Picard (2005) . As (one of) the first, Bailenson
et al. (2008) has shown that using both physio-
logical signals and facial expressions leads to bet-
ter ASP than using one of them; see also Section 1.
Hence, we advise to record three affective signals,
or have at least three features derived from them,
for each construct under investigation, in well con-
trolled research. Moreover, qualitative and subjective
measures should accompany the signals (e.g., ques-
tionnaires, video recordings, interviews, and Likert
scales); e.g., see (Hazlett and Hazlett, 1999; Healey
and Picard, 2005; Slater et al., 2006; Van den Broek,
2004; Westerink et al., 2008a). Please consult also
Section 3.1 on this topic.
3.3 A Physiology-driven Approach
A final prerequisite stems from the idea that ASP can
never be entirely based on psychological changes. As
discussed in Section 2, there are many factors out-
side one’s affective state that contaminate affective
signals. Beside validation and triangulation, another
way to deal with this is to employ a more physiology-
driven perspective (Tractinsky, 2004). Instead of ex-
pressing the goals of ASP directly in terms of affec-
tive states, they can often be stated in terms of the af-
fective signals themselves (Slater et al., 2006). For in-
stance, instead of inferring an air-trafficcontrollers or
driver’s stress level, thresholding SCL might be suffi-
cient; see also Box 1.
Note that there always remains an interpretation
in affective states. Then, the use of syntactic or struc-
tural pattern recognition for ASP should be (further)
explored. Its hierarchical approach to simplifying
complex patterns in affective signals is expected to be
valuable for ASP.
3.4 Contributions of Signal Processing
The majority of research on ASP is conducted by psy-
chology, physiology, medicine, human-computer in-
teraction, or artificial intelligence. Hence, true sig-
nal processing expertise is often missing; see also
Box 1. In particular, expertise from biomedical sig-
nal processing could significantly contribute to ASP’s
progress. We will now address a triplet of issues:
The development of filters tailored to the speci-
fications of ASP sensors and to ASP’s applications
could significantly boost the performance of ASP.
Affective signals are mostly sampled in a discrete
fashion. The required AD conversion, however, can
distort the signal; i.e., aliasing. For all possible sig-
nals, with all possible amplifiers and sensors, it is rec-
ommended to determine the relation between sample
frequency and signal loss / distortion. So far, this has
not been done and guidelines are provided founded
on weak assumptions; see also Box 1. Then, for all
signals, also the Nyquist frequency could be defined.
A benchmark should be founded with verified af-
fective signals. This would enable objective perfor-
mance measurements of signal processing and pattern
recognition techniques. The principle of triangulation
(see Section 3.2) could be applied using it and the
generic applicability of techniques could be tested.
Moreover, it could be used for concurrent validation
(see Section 3.1); e.g., through comparing different
signals or apparatus that can substitute each other.
This paper provided both an overview and a review
of ASP and explained the lack of success of ASP;
see Section 2 and Tables 1 and 2. Next, in Sec-
tion 3, four prerequisites are introduced from which
ASP is expected to benefit significantly: validation,
the principle of triangulation, and a physiological-
driven approach, and contributions of the signal pro-
cessing community. In addition, a real world case is
discussed in Box 1, which illustrated the use of the
proposed prerequisites.
With the guidelines provided and the future’s
progress ahead, we envision embedding of ASP in
various professional and consumer settings, as a key
factor of our every day life. A broad range of probes
have been developed over the years (Westerink et al.,
2008a), which illustrate the feasibility of embedding
ASP in various settings. Let us briefly denote three
of them: 1) For more than a decade, ASP is already
applied to determine the impact of advertisements on
people (Hazlett and Hazlett, 1999); for a review on
emotion measurement in advertising, see (Poels and
Dewitte, 2006). 2) Almost half a century ago, with
the development of Eliza (Weizenbaum, 1966) both
the possible implications of AI (for medicine) and the
limitations of (classic) AI became apparent. Among
many others, Liu et al. (2008), Slater et al. (2006),
and Van den Broek (2004) denote how ASP can help
AI to mature and to be of real value in this field. 3)
ASP has often been applied with pilots and in auto-
motive industry (Healey and Picard, 2005; Westerink
et al., 2008a), as is also denoted in Box 1. Healey and
Picard (2005) also provide a brief overview of litera-
ture on this topic.
With Ambient Intelligence evolving, (wireless)
sensor networks becoming more mature, and with
prerequisites that enable the exploitation of ASP’s
full potential, empathic machines should come within
reach. And would it not be an appealing idea to live
in an empathic surrounding that adapts to your mood
and emotions, which can calm you or help you to con-
centrate when required?
BIOSIGNALS 2009 - International Conference on Bio-inspired Systems and Signal Processing
Box 1: Application of the prerequisites in practice
One of the first large-scale real-world cases in which ASP is applied is the driving application of Healey and Picard (2005). They
applied ASP on ECG, EMG, EDA, and respiration to determine the stress of 24 participants, during at least 50 minutes real world
driving, completed by questionnaires and video recordings. Healey and Picard (2005) were able to distinguish between three stress
levels, using five minute time windows. In addition, through ASP, they developed a continuous stress metric. In this box, we apply
our ASP prerequisites for a critical analysis of this case; each prerequisite is denoted separately in relation to the case.
Validation. Section 3.1 denotes content, criteria-related, construct, and ecological validation. The continuous stress metric developed
by Healey and Picard predicts three situations: sitting still, driving on the highway, and driving in town. These situations are likely to
be confounded by movement. Movement, however, is not necessarily correlated with stress level. And even when movement reflects
a more excited state, it does not always come with a negative bias. As affective signals are also influenced by movement, this makes it
uncertain whether the actually measured construct is movement, stress, or a combination of both. This, together with a weak definition,
leads to a limited content validity.
For the continuous stress metric employed, Healey and Picard show a very high inter-observer reliability. Moreover, this stress
metric has a very high temporal resolution. Nonetheless, as the authors elaborate, this was in some cases not enough to deal with the
short latencies of some physiological signals. In addition, subjective stress reports could not be done during driving, so they were
conducted after the drive. As the drives were quite long, this might have lead to a decrease in postdictive validity. However, they were
congruent with averages of the continuous stress metric. Taken together, the criteria-related validity of the measurements is high.
Construct validation refers to a nomological framework built around the construct of interest. They nicely describe their stress
hypothesis in terms of the activity of the sympathetic and parasympathetic nervous systems. However, a description of the relation
between most of the physiological signals and these dimensions is only elaborated to alimited extent. Moreover, the HRVmeasurement
is questionable as it was not corrected for respiration (Grossman and Taylor, 2007). All in all, the construct validity is poor.
The ecological validity of Healey and Picard (2005) is excellent: the research results can be generalized to driving in general, most
likely to all driving situations. One could argue that the results are hard to generalize to other domains of application. This is indeed a
valid observation; however, the authors do (correctly) not claim to want so.
Triangulation. Healey and Picard (2005) use multiple signals for the construct under investigation, complemented by subjective
self-reports and observer scores. Healey and Picard correlated different physiological signals to see to what extent they describe the
same construct and used self-reports and observer scores to validate the physiological changes and, hence, created a higher validity;
see also Section 3.2. Hence, they successfully adopted the principle of triangulation.
Physiology-driven approach. Instead of continuously inferring the stress level of the driver, it might be sufficient to express a
high stress level in terms of one or a few physiological signals. For instance, (Healey and Picard, 2005) show that EDA is strongly
correlated with their continuous stress metric. So, using EDA as a stress scale might be sufficient for actual applications. By setting
some thresholds in the EDA, it can also be used for music selection or distraction management (e.g., cell phones), as the authors
propose. This reduces the amount of sensors and computing power needed, making it more feasible for practical application; see also
Section 3.3.
Signal processing contributions. Healey and Picard (2005) state that “Each signal was sampled at a rate appropriate for capturing
the information contained in the signal ...”. Such an ill specified statement is in line with other research where similar statements are
made or the subject is ignored completely. However, Healey and Picard (2005) continue with “ . . . constrained by the sampling rates
available ...”, which explains at least partly the reported sample frequencies; i.e., ECG: 496 Hz; EDA and respiration: 31 Hz; and
EMG: 15.5 Hz. These are not reported previously as standard frequencies, if reported at all.
Tailored filters as proposed in the current paper are absent. The only filter defined is the 0.5 seconds averaging filter. Why this is
applied is not reported. Possibly, more filters were applied; however, their specifications are omitted. On the one hand, this makes it
hard to reproduce the research; on the other hand, it illustrates the lack of attention for filtering.
On several occasions throughout the paper, comparable work on aircraft pilots was mentioned. It would be of interest to compare
the rich set of data Healey and Picard (2005) gathered with other data sets. Then, the robustness of the variety of signals, their features
and parameters could be fully accessed. To extend from a few single initiatives to a more general practice, a benchmark should be set
up; see also Section 3.4.
Conclusion. The work of Healey and Picard (2005) is already a showcase for triangulation. Their results illustrate that their case is
per excellence also suitable for a physiologically driven approach. An additional advantage of this direction is that it can be further
explored even without the preferred thorough theoretical framework. This would ease the way of their work to possible application
areas. Moreover, the applicability of the processing schemes proposed by Healey and Picard (2005) could be verified with future
benchmarks and the robustness of their approach could be strengthened through the usage of tailored filters and ASP techniques.
Hence, using the framework introduced in this paper, the case discussed could be substantially improved, even post-hoc, although its
foundation is already good and unique in its kind.
The authors would like to thank Marjolein van der
Zwaag and Stijn de Waele for their comments on an
earlier version of this paper. Furthermore, we would
like to thank the anonymous reviewers, who provided
us the opportunity to improve this paper.
Bailenson, J. N., Pontikakis, E. D., Mauss, I. B., Gross, J. J.,
Jabon, M. E., Hutcherson, C. A., Nass, C., and John,
O. (2008). Real-time classification of evoked emo-
tions using facial feature tracking and physiological
responses. International Journal of Human-Computer
Studies, 66(5):303–317.
Berntson, G. G., Bigger, J. T., Eckberg, D. L., Grossman,
P., Kaufmann, P. G., Malik, M., Nagaraja, H. N.,
Porges, S. W., Saul, J. P., Stone, P. H., and van der
Molen, M. W. (1997). Heart rate variability: Origins,
methods, and interpretive caveats. Psychophysiology,
Boucsein, W. (1992). Electrodermal activity. New York,
NY, USA: Plenum Press.
Cacioppo, J. and Tassinary, L. (1990). Inferring psychologi-
cal significance from physiological signals. American
Psychologist, 45(1):16–28.
Choi, A. and Woo, W. (2005). Physiological sensing and
feature extraction for emotion recognition by exploit-
ing acupuncture spots. Lecture Notes in Computer
Science (Affective Computing and Intelligent Interac-
tion), 3784:590–597.
Cowie, R., Douglas-Cowie, E., Tsapatsoulis, N., Votsis,
G., Kollias, S., Fellenz, W., and Taylor, J. G. (2001).
Emotion recognition in human–computer interaction.
IEEE Signal Processing Magazine, 18(1):32–80.
Duke, A. and Novicevic, M. M. (2008). Historical founda-
tions of social effectiveness? Dale Carnegie’s princi-
ples. Social Influences, 3(2):132–142.
Frederickson, B. L., Manusco, R. A., Branigan, C., and Tu-
gade, M. M. (2000). The undoing effect of positive
emotions. Motivation and Emotion, 24(4):237–257.
Gendolla, G. H. E. and Brinkman, K. (2005). The role
of mood states in self-regulation: Effects on action
preferences and resource mobilization. European Psy-
chologist, 10(3):187–198.
Grossman, P. and Taylor, E. W. (2007). Toward understand-
ing respiratory sinus arrhythmia: Relations to cardiac
vagal tone, evolution and biobehavioral functions. Bi-
ological Psychology, 74(2):263–285.
Haag, A., Goronzy, S., Schaich, P., and Williams, J. (2004).
Emotion recognition using bio-sensors: First steps to-
wards an automatic system. Lecture notes in computer
science (Affective Dialogue Systems), 3068:36–48.
Hazlett, R. L. and Hazlett, S. Y. (1999). Emotional response
to television commercials: Facial EMG vs. self-report.
Journal of Advertising Research, 39(2):7–23.
Healey, J. A. and Picard, R. W. (2005). Detecting stress dur-
ing real-world driving tasks using physiological sen-
sors. IEEE Transactions on Intelligent Transportation
Systems, 6(2):156–166.
Heath, L. (2001). Triangulation: Methodology, pages
15901–15906. Elsevier Science Ltd.: Oxford, UK,
1 edition. ISBN: 978-0-08-043076-8.
Katsis, C. D., Katertsidis, N., Ganiatsas, G., and Fotiadis,
D. I. (2008). Toward emotion recognition in car-racing
drivers: A biosignal processing approach. IEEE
Transactions on Systems, Man, and Cybernetics–Part
A: Systems and Humans, 38(3):502–512.
Kim, J. and Andr´e, E. (in press). Emotion recognition
based on physiological changes in music listening.
IEEE Transactions on Pattern Analysis Machine In-
Kim, K. H., Bang, S. W., and Kim, S. R. (2004). Emo-
tion recognition system using short-term monitoring
of physiological signals. Medical & Biological Engi-
neering & Computing, 42(3):419–427.
Leon, E., Clarke, G., Callaghan, V., and Sepulveda, F.
(2007). A user-independent real-time emotion recog-
nition system for software agents in domestic environ-
ments. Engineering Applications of Artificial Intelli-
gence, 20(3):337–345.
Lisetti, C. L. and Nasoz, F. (2004). Using noninvasive wear-
able computers to recognize human emotions from
physiological signals. EURASIP Journal on Applied
Signal Processing, 2004(11):1672–1687.
Liu, C., Conn, K., Sarkar, N., and Stone, W. (2008).
Physiology-based affect recognition for computer-
assisted intervention of children with Autism Spec-
trum Disorder. International Journal of Human-
Computer Studies, 66(9):662–677.
Minsky, M. (2006). The Emotion Machine: Commonsense
Thinking, Artificial Intelligence, and the Future of the
Human Mind. New York, NY, USA: Simon & Schus-
Nasoz, F., Alvarez, K., Lisetti, C. L., and Finkelstein, N.
(2003). Emotion recognition from physiological sig-
nals for presence technologies. International Journal
of Cognition, Technology and Work, 6:4–14.
Picard, R. W. (1997). Affective Computing. Boston MA,
USA: MIT Press.
Picard, R. W., Vyzas, E., and Healey, J. (2001). Toward
machine emotional intelligence: Analysis of affec-
tive physiological state. IEEE Transactions on Pat-
tern Analysis and Machine Intelligence, 23(10):1175–
Poels, K. and Dewitte, S. (2006). How to capture the heart?
reviewing 20 years of emotion measurement in adver-
tising. Journal of Advertising Research, 46(1):18–37.
Rani, P., Liu, C., Sarkar, N., and Vanman, E. (2006). An em-
pirical study of machine learning techniques for affect
recognition in human-robot interaction. Pattern Anal-
ysis & Applications, 9(1):58–69.
BIOSIGNALS 2009 - International Conference on Bio-inspired Systems and Signal Processing
Reaz, M. B. I., Hussain, M. S., and Mohd-Yasin, F. (2006).
Techniques of EMG signal analysis: detection, pro-
cessing, classification and applications. Biological
Procedures Online, 8(1):11–35.
Samboa, H., Silva, F., Silva, H., and ao, R. F. (2008). PLUX
– Biosignals Aquisition and Processing. URL:
[Last accessed on November 03,
Scheirer, J., Fernandez, R., Klein, J., and Picard, R. W.
(2002). Frustrating the user on purpose: a step to-
ward building an affective computer. Interacting with
Computers, 14(2):93–118.
Schuler, J. L. H. and O’Brien, W. H. (1997). Cardiovascu-
lar recovery from stress and hypertension factors: A
meta-analytic view. Psychophysiology, 34:649–659.
Sinha, R. and Parsons, O. A. (1996). Multivariate response
patterning of fear. Cognition and Emotion, 10(2):173–
Slater, M., Guger, C., Edlinger, G., Leeb, R., Pfurtscheller,
G., Antley, A., Garau, M., Brogni, A., and Friedman,
D. (2006). Analysis of physiological responses to a
social situation in an immersive virtual environment.
Presence: Teleoperators and Virtual Environments,
Takahashi, K. (2003). Remarks on emotion recognition
from bio-potential signals. In Proceedings of the IEEE
International Conference on Systems, Man and Cy-
bernetics, volume 2, pages 1655–1659, Palmerston
North, New Zealand.
Tractinsky, N. (2004). Tools over solutions? comments on
interacting with computers special issue on affective
computing. Interacting with Computers, 16(4):751–
Van den Broek, E. L. (2004). Emotional Prosody Measure-
ment (EPM): A voice-based evaluation method for
psychological therapy effectiveness. Studies in Health
Technology and Informatics (Medical and Care Com-
punetics), 103:118–125.
Wagner, J., Kim, J., and Andr´e, E. (2005). From physiologi-
cal signals to emotions: Implementing and comparing
selected methods for feature extraction and classifica-
tion. In Proceedings of the IEEE International Con-
ference on Multimedia and Expo (ICME).
Weizenbaum, J. (1966). ELIZA a computer program for
the study of natural language communication between
man and machine. Communications of the ACM,
Westerink, J. H. D. M., Ouwerkerk, M., overbeek, T.,
Pasveer, W. F., and de Ruyter, B. (2008a). Probing
Experiences: From Academic Research to Commer-
cial Propositions, volume 8 of Philips Research Book
Series. Springer: Dordrecht, The Netherlands.
Westerink, J. H. D. M., van den Broek, E. L., Schut,
M. H., van Herk, J., and Tuinenbreijer, K. (2008b).
Computing emotion awareness through galvanic skin
response and facial electromyography, volume 8 of
Philips Research Book Series, chapter 14 (Part II:
Probing in order to Feed Back), pages 137–150.
Springer: Dordrecht, The Netherlands.
Yannakakis, G. N. and Hallam, J. (2008). Entertain-
ment modeling through physiology in physical play.
International Journal of Human-Computer Studies,
Yoo, S. K., Lee, C. K., Park, J. Y., Kim, N. H., Lee, B. C.,
and Jeong, K. S. (2005). Neural network based emo-
tion estimation using heart rate variability and skin
resistance. Lecture Notes in Computer Science (Ad-
vances in Natural Computation), 3610:818–824.
Zeng, Z., Pantic, M., Roisman, G. I., and Huang, T. S.
(in press). A survey of affect recognition meth-
ods: Audio, visual, and spontaneous expressions.
IEEE Transactions on Pattern Analysis Machine In-
Zhai, J. and Barreto, A. (2006). Stress detection in com-
puter users through noninvasive monitoring of physi-
ological signals. Biomedical Science Instrumentation,