Skip to main content
SpringerLink
Log in
Book cover

IAPR Workshop on Multimodal Pattern Recognition of Social Signals in Human-Computer Interaction

MPRSS 2016: Multimodal Pattern Recognition of Social Signals in Human-Computer-Interaction pp 127–139Cite as

The SenseEmotion Database: A Multimodal Database for the Development and Systematic Validation of an Automatic Pain- and Emotion-Recognition System

  • Conference paper
  • First Online:

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 10183))

Abstract

In our modern industrial society the group of the older (generation 65+) is constantly growing. Many subjects of this group are severely affected by their health and are suffering from disability and pain. The problem with chronic illness and pain is that it lowers the patient’s quality of life, and therefore accurate pain assessment is needed to facilitate effective pain management and treatment. In the future, automatic pain monitoring may enable health care professionals to assess and manage pain in a more and more objective way. To this end, the goal of our SenseEmotion project is to develop automatic pain- and emotion-recognition systems for successful assessment and effective personalized management of pain, particularly for the generation 65+. In this paper the recently created SenseEmotion Database for pain- vs. emotion-recognition is presented. Data of 45 healthy subjects is collected to this database. For each subject approximately 30 min of multimodal sensory data has been recorded. For a comprehensive understanding of pain and affect three rather different modalities of data are included in this study: biopotentials, camera images of the facial region, and, for the first time, audio signals. Heat stimulation is applied to elicit pain, and affective image stimuli accompanied by sound stimuli are used for the elicitation of emotional states.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Notes

  1. 1.

    The IAPS identification numbers for the pleasant images were: 1650, 2216, 4311, 4611, 4658, 4659, 4664, 4676, 4677, 4690, 4694, 4695, 4800, 4810, 5460, 5470, 5626, 5629, 7502, 8030, 8080,8178, 8179, 8180, 8185, 8186, 8191, 8193, 8210, 8251, 8300, 8340, 8341, 8370, 8499, 8501. The identification numbers for the unpleasant images were: 1050, 1052, 1113, 1120, 1201, 1525, 1932, 2811, 3150, 3250, 3400, 3500, 5972, 6021, 6022, 6210, 6212, 6260, 6312, 6315, 6415, 6510, 6530, 6550, 6570, 6821, 8480, 8485, 9050, 9250, 9254, 9300, 9600, 9620, 9622, 9902, 9910, 9921. The identification number for the neutral images were: 5471, 5731, 6150, 7002, 7009, 7025, 7030, 7034, 7035, 7036, 7038, 7040, 7041, 7050, 7052, 7053, 7055, 7056, 7057, 7059, 7090, 7100, 7130, 7140, 7150, 7161, 7170, 7185, 7233, 7235, 7493, 7500, 7547, 7705. Mean valence and arousal ratings across image sets were: pleasant images (valance: M = 6.94, arousal: M = 6.40), unpleasant images (valance: M = 2.72, arousal: M = 6.42), and neutral images (valance: M = 5.03, arousal: M = 2.96).

  2. 2.

    The EmoPicS identification numbers for the pleasant images were: 006, 008, 028, 043, 050, 052, 053, 055, 056, 057, 058, 059, 061, 062, 063, 064, 065, 066, 067, 069, 070, 071, 075, 078. The identification numbers for the unpleasant images were: 207, 210, 211, 213, 214, 216, 219, 222, 229, 231, 232, 235, 238, 244, 250, 251, 252, 254, 321, 325, 326, 329. The identification numbers for the neutral images were: 123, 125, 127, 185, 188, 195, 196, 277, 281, 301, 302, 318, 335, 341, 342, 349, 354, 356, 365, 368, 371, 372, 373, 374, 376, 377. Mean valence and arousal ratings across image sets were: pleasant images (valance: M = 6.94, arousal: M = 5.57), unpleasant images (valance: M = 2.61, arousal: M = 6.26), and neutral images (valance: M = 5.02, arousal: M = 2.84).

  3. 3.

    http://www.uni-ulm.de/~hhoffman/emotions.

References

  1. Boiten, F.A., Frijda, N.H., Wientjes, C.J.E.: Emotions and respiratory patterns: review and critical analysis. Int. J. Psychophysiol. Official J. Int. Organ. Psychophysiol. 17(2), 103–128 (1994)

    Article  Google Scholar 

  2. Bradley, M.M., Codispoti, M., Cuthbert, B.N., Lang, P.J.: Emotion and motivation I: defensive and appetitive reactions in picture processing. Emotion (Washington, D.C.) 1(3), 276–298 (2001)

    Google Scholar 

  3. Bradley, M.M., Lang, P.J.: Measuring emotion: the self-assessment manikin and the semantic differential. J. Behav. Ther. Exp. Psychiatry 25(1), 49–59 (1994)

    Article  Google Scholar 

  4. Greenwald, M.K., Cook, E.W., Lang, P.J.: Affective judgment and psychophysiological response: dimensional covariation in the evaluation of pictorial stimuli. J. Psychophysiol. 3, 51–64 (1989)

    Google Scholar 

  5. Gruss, S.: Schmerzerkennung anhand psychophysiologischer Signale mithilfe maschineller Lerner. Dissertation, Universität Ulm (2015)

    Google Scholar 

  6. Gruss, S., Treister, R., Werner, P., Traue, H.C., Crawcour, S., Andrade, A., Walter, S.: Pain intensity recognition rates via biopotential feature patterns with support vector machines. PLoS ONE 10(10), 1–14 (2015)

    Article  Google Scholar 

  7. Haag, A., Goronzy, S., Schaich, P., Williams, J.: Emotion recognition using bio-sensors: first steps towards an automatic system. In: André, E., Dybkjær, L., Minker, W., Heisterkamp, P. (eds.) ADS 2004. LNCS, vol. 3068, pp. 36–48. Springer, Heidelberg (2004). doi:10.1007/978-3-540-24842-2_4

    Chapter  Google Scholar 

  8. Hammal, Z., Cohn, J.F.: Automatic detection of pain intensity. In: Proceedings of the 14th ACM International Conference on Multimodal Interaction, ICMI 2012, pp. 47–52. ACM, New York (2012)

    Google Scholar 

  9. Healey, J.: Physiological sensing of emotion. In: Calvo, R., D’Mello, S., Gratch, J., Kappas, A., (eds.) The Oxford Handbook of Affective Computing, pp. 204–216. Oxford University Press, New York (2015)

    Google Scholar 

  10. Jensen, C., Vasseljen, O., Westgaard, R.H.: The influence of electrode position on bipolar surface electromyogram recordings of the upper trapezius muscle. Eur. J. Appl. Physiol. 67(3), 266–273 (1993)

    Article  Google Scholar 

  11. Kächele, M., Amirian, M., Thiam, P., Werner, P., Walter, S., Palm, G., Schwenker, F.: Adaptive confidence learning for the personalization of pain intensity estimation systems. Evolving Syst. 8(1), 71–83 (2017)

    Article  Google Scholar 

  12. Kehlet, H.: Acute pain control and accelerated postoperative surgical recovery. Surg. Clin. North Am. 79(2), 431–443 (1999)

    Google Scholar 

  13. Kim, J., André, E.: Emotion recognition based on physiological changes in music listening. IEEE Trans. Pattern Anal. Mach. Intell. 30(12), 2067–2083 (2008)

    Article  Google Scholar 

  14. Lang, P.J.: Behavioral treatment and bio-behavioral assessment: Computer applications. In: Sidowski, J.B., Johnson, J.H., Williams, T.A. (eds.) Technology in Mental Health Care Delivery Systems, pp. 119–137. Ablex Publishing, Norwood (1980)

    Google Scholar 

  15. Lang, P.J.: The emotion probe: studies of motivation and attention. Am. Psychol. 50(5), 372–85 (1995)

    Article  Google Scholar 

  16. Lang, P.J., Bradley, M.M., Cuthbert, B.N.: International affective picture system (IAPS): Affective ratings of pictures and instruction manual. Technical report A-8, University of Florida, Gainesville, FL (2008)

    Google Scholar 

  17. Lang, P.J., Greenwald, M.K., Bradley, M.M., Hamm, A.O.: Looking at pictures: affective, facial, visceral, and behavioral reactions. Psychophysiology 30(3), 261–273 (1993)

    Article  Google Scholar 

  18. Lautenbacher, S.: Schmerzmessung. In: Basler, H.D., Franz, C., Kröner-Herwig, B., Rehfisch, H.P. (eds.) Psychologische Schmerztherapie, pp. 271–288. Springer, Berlin (2004)

    Google Scholar 

  19. Limbrecht-Ecklundt, K., Werner, P., Traue, H.C., Walter, S.: Mimische Aktivität differenzierter Schmerzintensitäten. Korrelation der Merkmale von Facial Action Coding System und Elektromyografie. Der. Schmerz 30(3), 248–256 (2016)

    Article  Google Scholar 

  20. McQuay, H., Moore, A., Justins, D.: Treating acute pain in hospital. Br. Med. J. 314(7093), 1531–1535 (1997)

    Article  Google Scholar 

  21. Meagher, M.W., Arnau, R.C., Rhudy, J.L.: Pain and emotion: effects of affective picture modulation. Psychosom. Med. 63(1), 79–90 (2001)

    Article  Google Scholar 

  22. Medoc advanced medical systems (2009)

    Google Scholar 

  23. Rhudy, J.L., Williams, A.E., McCabe, K.M., Nguyen, M.A., Rambo, P.: Affective modulation of nociception at spinal and supraspinal levels. Psychophysiology 42(5), 579–587 (2005)

    Google Scholar 

  24. Serpell, M.: Handbook of Pain Management. Springer, New York (2008)

    Book  Google Scholar 

  25. Tan, J.W., Andrade, A.O., Li, H., Walter, S., Hrabal, D., Rukavina, S., Limbrecht-Ecklundt, K., Hoffman, H., Traue, H.C.: Recognition of intensive valence and arousal affective states via facial electromyographic activity in young and senior adults. PLoS ONE 11(1), 1–14 (2016)

    Google Scholar 

  26. Wagner, J., Lingenfelser, F., Baur, T., Damian, I., Kistler, F., André, E.: The social signal interpretation (SSI) framework: multimodal signal processing and recognition in real-time. In Proceedings of the 21st ACM International Conference on Multimedia - MM 2013, pp. 831–834. ACM Press, New York (2013)

    Google Scholar 

  27. Walter, S., Gruss, S., Ehleiter, H., Tan, J., Traue, H.C., Crawcour, S., Werner, P., Al-Hamadi, A., Andrade, A.O., Moreira da Silva, G.: The BioVid heat pain database - data for the advancement and systematic validation of an automated pain recognition system. In: 2013 IEEE International Conference on Cybernetics (CYBCONF), pp. 128–131. IEEE, June 2013

    Google Scholar 

  28. Walter, S., Gruss, S., Limbrecht-Ecklundt, K., Traue, H.C., Werner, P., Al-Hamadi, A., Diniz, N., Moreira, G., Andrade, A.O.: Automatic pain quantification using autonomic parameters. Psychol. Neurosci. 7(3), 363–380 (2014)

    Article  Google Scholar 

  29. Werner, P., Al-Hamadi, A., Limbrecht-Ecklundt, K., Walter, S., Gruss, S., Traue, H.C.: Automatic pain assessment with facial activity descriptors. IEEE Trans. Affect. Comput. 99, 1–14 (2016)

    Article  Google Scholar 

  30. Wessa, M., Kanske, P., Neumeister, P., Bode, K., Heissler, J., Schönfelder, S.: EmoPics: Subjektive und psychophysiologische Evaluation neuen Bildmaterials für die klinisch-bio-psychologische Forschung. Zeitschrift für Klinische Psychologie und Psychotherapie 39(Suppl. 1/11), 77 (2010)

    Google Scholar 

  31. Wewers, M.E., Lowe, N.K.: A critical review of visual analogue scales in the measurement of clinical phenomena. Res. Nurs. Health 13(4), 227–236 (1990)

    Article  Google Scholar 

  32. Zwakhalen, S.M.G., Hamers, J.P.H., Abu-Saad, H.H., Berger, M.P.F.: Pain in elderly people with severe dementia: a systematic review of behavioural pain assessment tools. BMC Geriatr. 6(3), 1–15 (2006)

    Google Scholar 

Download references

Acknowledgment

This paper is based on work done within the project SenseEmotion funded by the German Federal Ministry of Education and Research (BMBF).

Author information

Authors and Affiliations

  1. University Clinic of Psychosomatic Medicine and Psychotherapy, Medical Psychology, Ulm University, 89075, Ulm, Germany

    Maria Velana, Sascha Gruss, Harald C. Traue & Steffen Walter

  2. Institute of Neural Information Processing, Ulm University, 89069, Ulm, Germany

    Georg Layher, Patrick Thiam, Yan Zhang, Viktor Kessler, Sascha Meudt, Heiko Neumann & Friedhelm Schwenker

  3. Department of Computer Science, Human-Centered Multimedia, University of Augsburg, 86159, Augsburg, Germany

    Daniel Schork, Jonghwa Kim & Elisabeth André

Authors
  1. Maria Velana
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sascha Gruss
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Georg Layher
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Patrick Thiam
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yan Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Daniel Schork
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Viktor Kessler
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Sascha Meudt
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Heiko Neumann
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Jonghwa Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Friedhelm Schwenker
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Elisabeth André
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Harald C. Traue
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. Steffen Walter
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Steffen Walter .

Editor information

Editors and Affiliations

  1. Universität Ulm, Ulm, Germany

    Friedhelm Schwenker

  2. Multimodal Communication and Computation, University of Southern California, Playa Vista, California, USA

    Stefan Scherer

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this paper

Cite this paper

Velana, M. et al. (2017). The SenseEmotion Database: A Multimodal Database for the Development and Systematic Validation of an Automatic Pain- and Emotion-Recognition System. In: Schwenker, F., Scherer, S. (eds) Multimodal Pattern Recognition of Social Signals in Human-Computer-Interaction. MPRSS 2016. Lecture Notes in Computer Science(), vol 10183. Springer, Cham. https://doi.org/10.1007/978-3-319-59259-6_11

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-59259-6_11

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-59258-9

  • Online ISBN: 978-3-319-59259-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation