Egon L. van den Broek, Joris H. Janssen, Joyce H. D. M. Westerink, Jennifer A. Healey



Although emotions are embraced by science, their recognition has not reached a satisfying level. Through a concise overview of affect, its signals, features, and classification methods, we provide understanding for the problems encountered. Next, we identify the prerequisites for successful Affective Signal Processing: validation (e.g., mapping of constructs on signals), triangulation, a physiology-driven approach, and contributions of the signal processing community. Using these directives, a critical analysis of a real-world case is provided. This illustrates that the prerequisites can become a valuable guide for Affective Signal Processing.


  1. Bailenson, J. N., Pontikakis, E. D., Mauss, I. B., Gross, J. J., Jabon, M. E., Hutcherson, C. A., Nass, C., and John, O. (2008). Real-time classification of evoked emotions using facial feature tracking and physiological responses. International Journal of Human-Computer Studies, 66(5):303-317.
  2. Berntson, G. G., Bigger, J. T., Eckberg, D. L., Grossman, P., Kaufmann, P. G., Malik, M., Nagaraja, H. N., Porges, S. W., Saul, J. P., Stone, P. H., and van der Molen, M. W. (1997). Heart rate variability: Origins, methods, and interpretive caveats. Psychophysiology, 34(6):623-648.
  3. Boucsein, W. (1992). Electrodermal activity. New York, NY, USA: Plenum Press.
  4. Cacioppo, J. and Tassinary, L. (1990). Inferring psychological significance from physiological signals. American Psychologist, 45(1):16-28.
  5. Choi, A. and Woo, W. (2005). Physiological sensing and feature extraction for emotion recognition by exploiting acupuncture spots. Lecture Notes in Computer Science (Affective Computing and Intelligent Interaction), 3784:590-597.
  6. Cowie, R., Douglas-Cowie, E., Tsapatsoulis, N., Votsis, G., Kollias, S., Fellenz, W., and Taylor, J. G. (2001). Emotion recognition in human-computer interaction. IEEE Signal Processing Magazine, 18(1):32-80.
  7. Duke, A. and Novicevic, M. M. (2008). Historical foundations of social effectiveness? Dale Carnegie's principles. Social Influences, 3(2):132-142.
  8. Frederickson, B. L., Manusco, R. A., Branigan, C., and Tugade, M. M. (2000). The undoing effect of positive emotions. Motivation and Emotion, 24(4):237-257.
  9. Gendolla, G. H. E. and Brinkman, K. (2005). The role of mood states in self-regulation: Effects on action preferences and resource mobilization. European Psychologist, 10(3):187-198.
  10. Grossman, P. and Taylor, E. W. (2007). Toward understanding respiratory sinus arrhythmia: Relations to cardiac vagal tone, evolution and biobehavioral functions. Biological Psychology, 74(2):263-285.
  11. Haag, A., Goronzy, S., Schaich, P., and Williams, J. (2004). Emotion recognition using bio-sensors: First steps towards an automatic system. Lecture notes in computer science (Affective Dialogue Systems), 3068:36-48.
  12. Hazlett, R. L. and Hazlett, S. Y. (1999). Emotional response to television commercials: Facial EMG vs. self-report. Journal of Advertising Research, 39(2):7-23.
  13. Healey, J. A. and Picard, R. W. (2005). Detecting stress during real-world driving tasks using physiological sensors. IEEE Transactions on Intelligent Transportation Systems, 6(2):156-166.
  14. Heath, L. (2001). Triangulation: Methodology, pages 15901-15906. Elsevier Science Ltd.: Oxford, UK, 1 edition. ISBN: 978-0-08-043076-8.
  15. Katsis, C. D., Katertsidis, N., Ganiatsas, G., and Fotiadis, D. I. (2008). Toward emotion recognition in car-racing drivers: A biosignal processing approach. IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans, 38(3):502-512.
  16. Kim, K. H., Bang, S. W., and Kim, S. R. (2004). Emotion recognition system using short-term monitoring of physiological signals. Medical & Biological Engineering & Computing, 42(3):419-427.
  17. Leon, E., Clarke, G., Callaghan, V., and Sepulveda, F. (2007). A user-independent real-time emotion recognition system for software agents in domestic environments. Engineering Applications of Artificial Intelligence, 20(3):337-345.
  18. Lisetti, C. L. and Nasoz, F. (2004). Using noninvasive wearable computers to recognize human emotions from physiological signals. EURASIP Journal on Applied Signal Processing, 2004(11):1672-1687.
  19. Liu, C., Conn, K., Sarkar, N., and Stone, W. (2008). Physiology-based affect recognition for computerassisted intervention of children with Autism Spectrum Disorder. International Journal of HumanComputer Studies, 66(9):662-677.
  20. Minsky, M. (2006). The Emotion Machine: Commonsense Thinking, Artificial Intelligence, and the Future of the Human Mind. New York, NY, USA: Simon & Schuster.
  21. Nasoz, F., Alvarez, K., Lisetti, C. L., and Finkelstein, N. (2003). Emotion recognition from physiological signals for presence technologies. International Journal of Cognition, Technology and Work, 6:4-14.
  22. Picard, R. W. (1997). Affective Computing. Boston MA, USA: MIT Press.
  23. Picard, R. W., Vyzas, E., and Healey, J. (2001). Toward machine emotional intelligence: Analysis of affective physiological state. IEEE Transactions on Pattern Analysis and Machine Intelligence, 23(10):1175- 1191.
  24. Poels, K. and Dewitte, S. (2006). How to capture the heart? reviewing 20 years of emotion measurement in advertising. Journal of Advertising Research, 46(1):18-37.
  25. Rani, P., Liu, C., Sarkar, N., and Vanman, E. (2006). An empirical study of machine learning techniques for affect recognition in human-robot interaction. Pattern Analysis & Applications, 9(1):58-69.
  26. Reaz, M. B. I., Hussain, M. S., and Mohd-Yasin, F. (2006). Techniques of EMG signal analysis: detection, processing, classification and applications. Biological Procedures Online, 8(1):11-35.
  27. Samboa, H., Silva, F., Silva, H., and ao, R. F. (2008). PLUX - Biosignals Aquisition and Processing. URL: http: // [Last accessed on November 03, 2008].
  28. Scheirer, J., Fernandez, R., Klein, J., and Picard, R. W. (2002). Frustrating the user on purpose: a step toward building an affective computer. Interacting with Computers, 14(2):93-118.
  29. Schuler, J. L. H. and O'Brien, W. H. (1997). Cardiovascular recovery from stress and hypertension factors: A meta-analytic view. Psychophysiology, 34:649-659.
  30. Sinha, R. and Parsons, O. A. (1996). Multivariate response patterning of fear. Cognition and Emotion, 10(2):173- 198.
  31. Slater, M., Guger, C., Edlinger, G., Leeb, R., Pfurtscheller, G., Antley, A., Garau, M., Brogni, A., and Friedman, D. (2006). Analysis of physiological responses to a social situation in an immersive virtual environment. Presence: Teleoperators and Virtual Environments, 15(5):553-569.
  32. Takahashi, K. (2003). Remarks on emotion recognition from bio-potential signals. In Proceedings of the IEEE International Conference on Systems, Man and Cybernetics, volume 2, pages 1655-1659, Palmerston North, New Zealand.
  33. Tractinsky, N. (2004). Tools over solutions? comments on interacting with computers special issue on affective computing. Interacting with Computers, 16(4):751- 757.
  34. Van den Broek, E. L. (2004). Emotional Prosody Measurement (EPM): A voice-based evaluation method for psychological therapy effectiveness. Studies in Health Technology and Informatics (Medical and Care Compunetics), 103:118-125.
  35. Wagner, J., Kim, J., and André, E. (2005). From physiological signals to emotions: Implementing and comparing selected methods for feature extraction and classification. In Proceedings of the IEEE International Conference on Multimedia and Expo (ICME).
  36. Weizenbaum, J. (1966). ELIZA - a computer program for the study of natural language communication between man and machine. Communications of the ACM, 9(1):36-45.
  37. Westerink, J. H. D. M., Ouwerkerk, M., overbeek, T., Pasveer, W. F., and de Ruyter, B. (2008a). Probing Experiences: From Academic Research to Commercial Propositions, volume 8 of Philips Research Book Series. Springer: Dordrecht, The Netherlands.
  38. Westerink, J. H. D. M., van den Broek, E. L., Schut, M. H., van Herk, J., and Tuinenbreijer, K. (2008b). Computing emotion awareness through galvanic skin response and facial electromyography, volume 8 of Philips Research Book Series, chapter 14 (Part II: Probing in order to Feed Back), pages 137-150. Springer: Dordrecht, The Netherlands.
  39. Yannakakis, G. N. and Hallam, J. (2008). Entertainment modeling through physiology in physical play. International Journal of Human-Computer Studies, 66(10):741-755.
  40. Yoo, S. K., Lee, C. K., Park, J. Y., Kim, N. H., Lee, B. C., and Jeong, K. S. (2005). Neural network based emotion estimation using heart rate variability and skin resistance. Lecture Notes in Computer Science (Advances in Natural Computation), 3610:818-824.
  41. Zhai, J. and Barreto, A. (2006). Stress detection in computer users through noninvasive monitoring of physiological signals. Biomedical Science Instrumentation, 42:495-500.

Paper Citation

in Harvard Style

L. van den Broek E., H. Janssen J., H. D. M. Westerink J. and A. Healey J. (2009). PREREQUISITES FOR AFFECTIVE SIGNAL PROCESSING (ASP) . In Proceedings of the International Conference on Bio-inspired Systems and Signal Processing - Volume 1: BIOSIGNALS, (BIOSTEC 2009) ISBN 978-989-8111-65-4, pages 426-433. DOI: 10.5220/0001780504260433

in Bibtex Style

author={Egon L. van den Broek and Joris H. Janssen and Joyce H. D. M. Westerink and Jennifer A. Healey},
booktitle={Proceedings of the International Conference on Bio-inspired Systems and Signal Processing - Volume 1: BIOSIGNALS, (BIOSTEC 2009)},

in EndNote Style

JO - Proceedings of the International Conference on Bio-inspired Systems and Signal Processing - Volume 1: BIOSIGNALS, (BIOSTEC 2009)
SN - 978-989-8111-65-4
AU - L. van den Broek E.
AU - H. Janssen J.
AU - H. D. M. Westerink J.
AU - A. Healey J.
PY - 2009
SP - 426
EP - 433
DO - 10.5220/0001780504260433