Skip to main content
SpringerLink
Log in
Book cover

International Workshop on Human Behavior Understanding

HBU 2012: Human Behavior Understanding pp 52–64Cite as

Using Speech Data to Recognize Emotion in Human Gait

  • Conference paper

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 7559))

Abstract

Robots that can recognize emotions can improve humans’ mental health by providing empathy and social communication. Emotion recognition by robots is challenging because unlike in human-computer environments, facial information is not always available. Instead, our method proposes using speech and gait analysis to recognize human emotion. Previous research suggests that the dynamics of emotional human speech also underlie emotional gait (walking). We investigate the possibility of combining these two modalities via perceptually common parameters: Speed, Intensity, irRegularity, and Extent (SIRE). We map low-level features to this 4D cross-modal emotion space and train a Gaussian Mixture Model using independent samples from both voice and gait. Our results show that a single, modality-mixed trained model can perform emotion recognition for both modalities. Most interestingly, recognition of emotion in gait using a model trained uniquely on speech data gives comparable results to a model trained on gait data alone, providing evidence for a common underlying model for emotion across modalities.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   49.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Clynes, M.: Sentics: The Touch of the Emotions. Prism Press, UK (1989)

    Google Scholar 

  2. Pollick, F.E., Paterson, H.M., Bruderlin, A., Sanford, A.J.: Perceiving affect from arm movement. J. Personal. 82, 51–61 (2001)

    Google Scholar 

  3. Van Bezooijen, R., Van Otto, S.A., Heenan, T.A.: Recognition of vocal dimensions of emotion: A three-nation study to identify universal characteristics. J. Cross-Cultural Psych. 14, 387–406 (1983)

    Article  Google Scholar 

  4. Juslin, P.N., Laukka, P.: Communication of emotions in vocal expression and music performance: different channels, same code? Psychol. Bull. 129(5), 770–814 (2003)

    Article  Google Scholar 

  5. Spencer, H.: The origin and function of music. Fraser’s Magazine 56, 396–408 (1857)

    Google Scholar 

  6. Scherer, K.H.: Vocal affect expression: A review and a model for future research. Psychol. Bull. 99, 143–165 (1986)

    Article  Google Scholar 

  7. Snowdon, C.T.: Expression of emotion in non-human animals. In: Davidson, R.J., Sherer, K.H., Goldsmith, H.H. (eds.) Handbook of affective sciences, pp. 457–480. Oxford University Press, London (2003)

    Google Scholar 

  8. Breazeal, C.: Designing sociable robots, 1st edn. The MIT Press, Cambridge (2004)

    Google Scholar 

  9. Lim, A., Ogata, T., Okuno, H.G.: Towards expressive musical robots: a cross-modal framework for emotional gesture, voice and music. EURASIP J. Audio, Speech, and Music Proc. 2012(3) (2012)

    Google Scholar 

  10. Lim, A., Ogata, T., Okuno, H.G.: Converting emotional voice to motion for robot telepresence. In: Humanoids, Bled, pp. 472–479 (2011)

    Google Scholar 

  11. Cowie, R., et al.: Emotion recognition in human-computer interaction. IEEE Signal Proc. Magazine 18(1), 32–80 (2001)

    Article  Google Scholar 

  12. Fernandez, R., Picard, R.W.: Classical and Novel Discriminant Features for Affect Recognition from Speech. In: INTERSPEECH, pp. 4–8 (2005)

    Google Scholar 

  13. Mion, L., De Poli, G.: Score-independent audio features for description of music expression. IEEE Trans. Audio Speech Lang. Process. 16(2), 458–466 (2008)

    Article  Google Scholar 

  14. Livingstone, S.R., Brown, A.R., Muhlberger, R., Thompson, W.F.: Modifying score and performance changing musical emotion: a computational rule system for modifying score and performance. Comput. Music J. 34(1), 41–65 (2010)

    Article  Google Scholar 

  15. Amaya, K., Bruderlin, A., Calvert, T.: Emotion from motion. Graph. In: Interface, pp. 222–229 (1996)

    Google Scholar 

  16. Pelachaud, C.: Studies on gesture expressivity for a virtual agent. Speech Commun. 51(7), 630–639 (2009)

    Article  Google Scholar 

  17. Camurri, A., Volpe, G.: Communicating expressiveness and affect in multimodal interactive systems. Multimedia 12(1), 43–53 (2005)

    Article  Google Scholar 

  18. Douglas-Cowie, E., Cowie, R., Sneddon, I., Cox, C., Lowry, O., McRorie, M., Martin, J.-C., Devillers, L., Abrilian, S., Batliner, A., Amir, N., Karpouzis, K.: The HUMAINE Database: Addressing the Collection and Annotation of Naturalistic and Induced Emotional Data. In: Paiva, A.C.R., Prada, R., Picard, R.W. (eds.) ACII 2007. LNCS, vol. 4738, pp. 488–500. Springer, Heidelberg (2007)

    Chapter  Google Scholar 

  19. Roether, C.L., Omlor, L., Christensen, A., Giese, M.A.: Critical features for the perception of emotion from gait. J. Vision 9(6), 15, 1–32 (2009)

    Article  Google Scholar 

  20. Montepare, J.M., Goldstein, S.B.: The identification of emotions from gait information. J. Nonverbal Behav. 11(1), 33–42 (1987)

    Article  Google Scholar 

  21. Janssen, D., et al.: Recognition of emotions in gait patterns by means of artificial neural nets. J. Nonverbal Behav. 32, 79–92 (2008)

    Article  Google Scholar 

  22. Unuma, M., Anjyo, K., Takeuchi, R.: Fourier principles for emotion-based human figure animation. In: SIGGRAPH, Los Angeles, pp. 91–96 (1995)

    Google Scholar 

  23. Montepare, J., Koff, E., Zaichik, D., Albert, M.: The use of body movements and gestures as cues to emotions in younger and older adults. J. Nonverbal Behav. 23(2), 133–152 (1999)

    Article  Google Scholar 

  24. Karg, M., Kuhnlenz, K., Buss, M.: Recognition of affect based on gait patterns. IEEE Trans. Sys., Man, Cyber. 40(4), 1050–1061 (2010)

    Article  Google Scholar 

  25. Ma, Y., Paterson, H.M., Pollick, F.E.: A motion-capture library for the study of identity, gender, and emotion perception from biological motion. Behav. Res. Meth., Inst., & Comp. 38, 134–141 (2006)

    Article  Google Scholar 

  26. Bernhardt, D.: Detecting emotions from everyday body movements. Presenccia PhD Sym., Barcelona (2007)

    Google Scholar 

  27. Pedregosa, F., et al.: Scikit-learn: Machine Learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011)

    MathSciNet  Google Scholar 

  28. Parrot, W.G.: Emotions in social psychology. Philadelphia Press, Philadelphia (2001)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Graduate School of Informatics, Kyoto University, Sakyo-ku, Kyoto, Japan, 606-8501

    Angelica Lim & Hiroshi G. Okuno

Authors
  1. Angelica Lim
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hiroshi G. Okuno
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Department of Computer Engineering, Boğaziçi University, 34342, Bebek, Istanbul, Turkey

    Albert Ali Salah

  2. Department of Electric Engineering, Universidad de Chile, Av. Tupper 2007, Santiago, Chile

    Javier Ruiz-del-Solar

  3. Computer Science Department, Carnegie Mellon University, 15213, Pittsburgh, PA, USA

    Çetin Meriçli

  4. FLOWERS Research Team, INRIA Bordeaux Sud-Ouest, 33405, Talence, Cedex, France

    Pierre-Yves Oudeyer

Rights and permissions

Reprints and permissions

Copyright information

© 2012 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Lim, A., Okuno, H.G. (2012). Using Speech Data to Recognize Emotion in Human Gait. In: Salah, A.A., Ruiz-del-Solar, J., Meriçli, Ç., Oudeyer, PY. (eds) Human Behavior Understanding. HBU 2012. Lecture Notes in Computer Science, vol 7559. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-34014-7_5

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-34014-7_5

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-34013-0

  • Online ISBN: 978-3-642-34014-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation