Multiresolution Feature Extraction During Psychophysiological Inference: Addressing Signals Asynchronicity

  • François CourtemancheEmail author
  • Aude Dufresne
  • Élise L. LeMoyne
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8908)


Predicting the psychological state of the user using physiological measures is one of the main objectives of physiological computing. While numerous works have addressed this task with great success, a large number of challenges remain to be solved in order to develop recognition approaches that can precisely and reliably feed human-computer interaction systems. This chapter focuses on one of these challenges which is the temporal asynchrony between different physiological signals within one recognition model. The chapter proposes a flexible and suitable method for feature extraction based on empirical optimisation of windows’ latency and duration. The approach is described within the theoretical framework of the psychophysiological inference and its common implementation using machine learning. The method has been experimentally validated (46 subjects) and results are presented. Empirically optimised values for the extraction windows are provided.


Affective signal processing Temporal construction Psychophysiological inference Triangulation 



This work was supported by NSERC (Natural Sciences and Engineering Research Council of Canada), the Canadian Space Agency and Bell Canada. The authors would like to thank the Bell Web Solutions User Experience Center for providing the eye-tracker system used in this research. We also wish to thank Laurence Dumont for early comments on the manuscript.


  1. 1.
    Ellsworth, P.C.: William James and emotion: Is a century of fame worth a century of misunderstanding? Psychol. Rev. 101(2), 222–229 (1994)CrossRefGoogle Scholar
  2. 2.
    Allanson, J., Fairclough, S.H.: A research agenda for physiological computing. Interact. Comput. 16(5), 857–878 (2004)CrossRefGoogle Scholar
  3. 3.
    Cacioppo, J.T., Tassinary, L.G.: Inferring psychological significance from physiological signals. Am. Psychol. 45(1), 16–28 (1990)CrossRefGoogle Scholar
  4. 4.
    Cacioppo, J.T., Tassinary, L.T., Berntson, G., et al.: Psychophysiological science: interdisciplinary approaches to classic questions about the mind. In: Cacioppo, J.T., Tassinary, L.G., Berntson, G. (eds.) Handbook of Psychophysiology, pp. 1–18. Cambride University Press, New York (2007)CrossRefGoogle Scholar
  5. 5.
    Kreibig, S.D.: Autonomic nervous system activity in emotion: a review. Biol. Psychol. 84(3), 394–421 (2010)CrossRefGoogle Scholar
  6. 6.
    Bamidis, P., et al.: An integrated approach to emotion recognition for advanced emotional intelligence. In: Jacko, J. (ed.) Human-Computer Interaction. Ambient, Ubiquitous and Intelligent Interaction, pp. 565–574. Springer, Berlin (2009)CrossRefGoogle Scholar
  7. 7.
    Chanel, G., et al.: Short-term emotion assessment in a recall paradigm. Int. J. Hum Comput Stud. 67(8), 607–627 (2009)CrossRefGoogle Scholar
  8. 8.
    Christie, I.C., Friedman, B.H.: Autonomic specificity of discrete emotion and dimensions of affective space: a multivariate approach. Int. J. Psychophysiol. 51(2), 143–153 (2004)CrossRefGoogle Scholar
  9. 9.
    Haag, A., Goronzy, S., Schaich, P., Williams, J.: Emotion recognition using bio-sensors: first steps towards an automatic system. In: André, E., Dybkjær, L., Minker, W., Heisterkamp, P. (eds.) ADS 2004. LNCS (LNAI), vol. 3068, pp. 36–48. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  10. 10.
    Kolodyazhniy, V., et al.: An affective computing approach to physiological emotion specificity: toward subject-independent and stimulus-independent classification of film-induced emotions. Psychophysiology 48(7), 908–922 (2011)CrossRefGoogle Scholar
  11. 11.
    Picard, R.W., Vyzas, E., Healey, J.: Toward machine emotional intelligence: analysis of affective physiological state. IEEE Trans. Pattern Anal. Mach. Intell. 23(10), 1175 (2001)CrossRefGoogle Scholar
  12. 12.
    Verhoef, T., et al.: Bio-sensing for emotional characterization without word labels. In: Jacko, J. (ed.) Human-Computer Interaction. Ambient, Ubiquitous and Intelligent Interaction, pp. 693–702. Springer, Berlin (2009)CrossRefGoogle Scholar
  13. 13.
    van den Broek, E., et al.: Prerequisites for affective signal processing (ASP) - Part III. In: Third International Conference on Bio-Inspired Systems and Signal Processing, Biosignals 2010. Valencia, Spain (2010)Google Scholar
  14. 14.
    van den Broek, E., et al.: Prerequisites for affective signal processing (ASP). In: International Conference on Bio-inspired Systems and Signal Processing. INSTICC Press, Portugal (2009)Google Scholar
  15. 15.
    van den Broek, E., et al.: Prerequisites for affective signal processing (ASP) - Part IV. In: 1st International Workshop on Bio-Inspired Human-Machine Interfaces and Healthcare Applications - B-Interface 2010, pp. 59–66. Valencia, Spain 2010Google Scholar
  16. 16.
    Cowie, R., et al.: Issues in data collection. In: Cowie, R., Pelachaud, C., Petta, P. (eds.) Emotion-Oriented Systems, pp. 197–212. Springer, Berlin (2011)CrossRefGoogle Scholar
  17. 17.
    Pizzagalli, D.A., et al.: Electoencephalography and high-density electrophysiological source localisation. In: Cacioppo, J.T., Tassinary, L.G., Bernston, G.G. (eds.) Handbook of Psychophysiology, pp. 56–84. Cambride University Press, New York (2007)CrossRefGoogle Scholar
  18. 18.
    Guyon, I., Elisseeff, A.: An introduction to variable and feature selection. J. Mach. Learn. Res. 3, 1157–1182 (2003)zbMATHGoogle Scholar
  19. 19.
    Rani, P., et al.: An empirical study of machine learning techniques for affect recognition in human–robot interaction. Pattern Anal. Appl. 9(1), 58–69 (2006)CrossRefMathSciNetGoogle Scholar
  20. 20.
    Zhihong, Z., et al.: A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE Trans. Pattern Anal. Mach. Intell. 31(1), 39–58 (2009)CrossRefGoogle Scholar
  21. 21.
    Bishop, C.M.: Pattern Recognition and Machine Learning. Springer, New York (2006)zbMATHGoogle Scholar
  22. 22.
    Kim, J., Elesabeth, A., Thurid, V.: Towards user-independent classification of multimodal emotional signals. In: 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops (ACII 2009) (2009)Google Scholar
  23. 23.
    AlZoubi, O., D’Mello, S.K., Calvo, R.A.: Detecting naturalistic expressions of nonbasic affect using physiological signals. IEEE Trans. Affect. Comput. 3(3), 298–310 (2012)CrossRefGoogle Scholar
  24. 24.
    Schuster, T., et al.: EEG-based valence recognition: what do we know about the influence of individual specificity? In: The Fourth International Conference on Advanced Cognitive Technologies and Applications (COGNITIVE 2012). Nice, France (2012)Google Scholar
  25. 25.
    Wilder, J.: Modern psychophysiology and the law of initial value. Am. J. Psychother. 12, 199–221 (1958)Google Scholar
  26. 26.
    Jennings, L.R., Gianaros, P.J., et al.: Methodology. In: Cacioppo, J.T., Tassinary, L.G., Bernston, G.G. (eds.) Handbook of Psychophysiology, pp. 812–833. Cambride University Press, New York (2007)CrossRefGoogle Scholar
  27. 27.
    Gunes, H., Pantic, M.: Automatic Measurement of Affect in Dimensional and Continuous Spaces: Why, What, and How?. In: Spink, A.J., et al. (eds.) 7th International Conference on Methods and Techniques in Behavioral Research, Measuring Behavior 2010, pp. 122–126. Noldus, Eindhoven (2010)Google Scholar
  28. 28.
    van der Zwaag, M.D., van den Broek, E., Janssen, J.H.: Guidelines for biosignal driven HCI. In: ACM CHI2010 workshop - Brain, Body, and Bytes: Physiological User Interaction. Atlanta, GA, USA (2010)Google Scholar
  29. 29.
    Coan, J.A., Allen, J.J.B.: Frontal EEG asymmetry as a moderator and mediator of emotion. Biol. Psychol. 67(1–2), 7–50 (2004)CrossRefGoogle Scholar
  30. 30.
    Lang, P.J., Bradley, M.M., Cuthbert, B.N: International affective picture system (IAPS): Affective ratings of pictures and instruction manual. Technical report B-3. 2008, University of Florida, Gainesville, FI (2008)Google Scholar
  31. 31.
    Bradley, M.M., Lang, P.J.: The International Affective Digitized Sounds (2nd edn., IADS-2): Affective Ratings of Sounds and Instruction Manual. Technical report B-3, University of Florida, Gainesville, FI (2007)Google Scholar
  32. 32.
    Anttonen, J., Surakka, V.: Emotions and heart rate while sitting on a chair. In: SIGCHI Conference on Human Factors in Computing Systems, pp. 491–499. ACM, Portland (2005)Google Scholar
  33. 33.
    Mühl, C., Heylen, D.: Cross-modal elicitation of affective experience. In: International Conference on Affective Computing and Intelligent Interaction and Workshops (ACII 2009). Workshop on Affective Brain-Computer Interfaces. Amsterdam, The Netherlands (2009)Google Scholar
  34. 34.
    Bradley, M.M., Lang, P.J.: Measuring emotion: The self-assessment manikin and the semantic differential. J. Behav. Ther. Exp. Psychiatry 25(1), 49–59 (1994)CrossRefGoogle Scholar
  35. 35.
    van den Broek, E.: Ubiquitous emotion-aware computing. Pers. Ubiquit. Comput. 17, 1–15 (2011)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  • François Courtemanche
    • 1
    Email author
  • Aude Dufresne
    • 2
  • Élise L. LeMoyne
    • 1
  1. 1.Tech3LabMontréalCanada
  2. 2.Department of CommunicationUniversity of MontréalMontréalCanada

Personalised recommendations