Abstract
In order to build artificial conversational interfaces that display behaviors that are credible and expressive, we should endow them with the capability to recognize, adapt to, and render emotion. In this chapter, we explain how the recognition of emotional aspects is managed within conversational interfaces, including modeling and representation, emotion recognition from physiological signals, acoustics, text, facial expressions, and gestures and how emotion synthesis is managed through expressive speech and multimodal embodied agents. We also cover the main open tools and databases available for developers wishing to incorporate emotion into their conversational interfaces.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
https://www.w3.org/TR/emotionml/. Accessed February 27, 2016.
- 2.
http://www.w3.org/TR/emotion-voc/. Accessed March 1, 2016.
- 3.
https://www.informatik.uni-augsburg.de/de/lehrstuehle/hcm/projects/tools/aubt/. Accessed February 27, 2016.
- 4.
http://compare.openaudio.eu/. Accessed February 27, 2016.
- 5.
http://www.fon.hum.uva.nl/praat/. Accessed February 27, 2016.
- 6.
https://uk.groups.yahoo.com/neo/groups/praat-users. Accessed February 27, 2016.
- 7.
http://stackoverflow.com/questions/tagged/praat. Accessed February 27, 2016.
- 8.
http://hcm-lab.de/projects/ssi/. Accessed February 27, 2016.
- 9.
http://www.audeering.com/research/opensmile. Accessed February 27, 2016.
- 10.
http://emotion-research.net/toolbox/toolbox_query_view?category=Database. Accessed February 27, 2016.
- 11.
http://catalog.elra.info/. Accessed February 27, 2016.
- 12.
http://www.nltk.org/. Accessed February 27, 2016.
- 13.
https://opennlp.apache.org/. Accessed February 27, 2016.
- 14.
http://sentiwordnet.isti.cnr.it/. Accessed February 27, 2016.
- 15.
http://csea.phhp.ufl.edu/media.html#bottommedia. Accessed February 27, 2016.
- 16.
https://www.cs.uic.edu/~liub/FBS/sentiment-analysis.html. Accessed February 27, 2016.
- 17.
http://www.wjh.harvard.edu/~inquirer/. Accessed February 27, 2016.
- 18.
http://www3.nd.edu/~mcdonald/Word_Lists.html. Accessed February 27, 2016.
- 19.
http://www.anvil-software.org/. Accessed February 27, 2016.
- 20.
- 21.
http://www.pitt.edu/~emotion/ck-spread.htm. Accessed February 27, 2016.
- 22.
http://mmifacedb.eu/. Accessed February 27, 2016.
- 23.
http://emofilt.syntheticspeech.de/. Accessed February 27, 2016.
- 24.
The example code shown is from the SEMAINE project: http://semaine.opendfki.de/wiki/FML. Accessed February 27, 2016.
- 25.
http://secom.ru.is/fml/. Accessed February 27, 2016.
- 26.
http://alma.dfki.de. Accessed February 27, 2016.
- 27.
http://sourceforge.net/projects/fearnot/files/FAtiMA/FAtiMA/. Accessed February 27, 2016.
- 28.
http://www.affective-sciences.org/content/exploring-your-ec. Accessed February 27, 2016.
- 29.
http://sentiment.christopherpotts.net/. Accessed February 27, 2016.
References
Allwood J, Cerrato L, Jokinen K, Naravetta C, Paggio P (2008) The MUMIN coding scheme for the annotation of feedback, turn management and sequencing phenomena. Lang Resour Eval 41(3/4):273–287. doi:10.1007/s10579-007-9061-5
Baccianella S, Esuli A, Sebastiani F (2010) SentiWordNet 3.0: an enhanced lexical resource for sentiment analysis and opinion mining. In: International conference on language resources and evaluation (LREC2010). European Language Resources Association (ELRA), Malta, 17–23 May 2010, pp 2200–2204
Batliner A, Schuller B, Seppi D, Steidl S, Devilliers L, Vidrascu L, Vogt T, Aharonson V, Amir N (2011) The automatic recognition of emotions in speech. In: Cowie R, Pelachaud C, Petta P (eds) Emotion-oriented systems. Springer Berlin Heidelberg, Berlin, Heidelberg, pp 71–99. doi:10.1007/978-3-642-15184-2_6
Ben Mimoun MS, Poncin I, Garnier M (2012) Case study—embodied virtual agents: an analysis on reasons for failure. J Retail Consum Serv 19(6):605–612. doi:10.1016/j.jretconser.2012.07.006
Boersma P, Weenink D (2016) Praat: doing phonetics by computer. http://www.fon.hum.uva.nl/praat/
Bos DO (2006) EEG-based emotion recognition; the influence of visual and auditory stimuli. http://hmi.ewi.utwente.nl/verslagen/capita-selecta/CS-Oude_Bos-Danny.pdf
Burkhardt F (2005) Emofilt: the simulation of emotional speech by prosody-transformation. In: Proceedings of the 9th European conference on speech communication and technology (Interspeech2005), Lisbon, Portugal, 4–8 Sept 2005, pp 509–512. http://www.isca-speech.org/archive/interspeech_2005/i05_0509.html
Cafaro A, Vilhjálmsson HH, Bickmore T, Heylen D, Pelachaud C(2014) Representing communicative functions in SAIBA with a unified function markup language. In: Bickmore T, Marsella S, Sidner C (eds) Intelligent virtual agents. Springer International Publishing, Switzerland, pp 81–94. doi:10.1007/978-3-319-09767-1_11
Callejas Z, Griol D, López-Cózar R (2011) Predicting user mental states in spoken dialogue systems. EURASIP J Adv Signal Process 1:6. doi:10.1186/1687-6180-2011-6
Calvo RA, D’Mello S, Gratch J, Kappas A (eds) (2014) The Oxford handbook of affective computing, 1st edn. Oxford University Press, Oxford. doi:10.1093/oxfordhb/9780199942237.001.0001
Castellano G, Leite I, Pereira A, Martinho C, Paiva A, McOwan PW (2010) Affect recognition for interactive companions: challenges and design in real world scenarios. J Multimodal User Interfaces 3(1–2):89–98. doi:10.1007/s12193-009-0033-5
Clavel C, Callejas Z (2016) Sentiment analysis: from opinion mining to human-agent interaction. IEEE Trans Affect Comput 7(1):74–93. doi:10.1109/TAFFC.2015.2444846
Cowie R, Cornelius R (2003) Describing the emotional states that are expressed in speech. Speech Commun 40(1–2):5–32. doi:10.1016/S0167-6393(02)00071-7
Ekman P (1999) Basic emotions. In: Dalgleish T, Power MJ (eds) Handbook of cognition and emotion. Wiley, Chichester, pp 45–60. doi:10.1002/0470013494.ch3
Ekman P (2003) Emotions revealed: recognizing faces and feelings to improve communication and emotional life, 1st edn. Times Books, New York
Ekman P, Rosenberg EL (eds) (2005) What the face reveals: basic and applied studies of spontaneous expression using the facial action coding system (FACS), 2nd edn. Oxford University Press, Oxford. doi:10.1093/acprof:oso/9780195179644.001.0001
Eyben F, Weninger F, Gross F, Schuller B (2013) Recent developments in openSMILE, the munich open-source multimedia feature extractor. In: Proceedings of the 21st ACM international conference on multimedia (MM’13), Barcelona, Spain, 21–25 Oct 2013, pp 835–838. doi:10.1145/2502081.2502224
Feldman R (2013) Techniques and applications for sentiment analysis. Commun ACM 56(4):82. doi:10.1145/2436256.2436274
Govind D, Prasanna SRM (2012) Expressive speech synthesis: a review. IJST 16(2):237–260. doi:10.1007/s10772-012-9180-2
Gratch J, Marsella S (eds) (2013) Social emotions in nature and artifact. Oxford University Press, Oxford 10.1093/acprof:oso/9780195387643.001.0001
Haag A, Goronzy S, Schaich P, Williams J (2004) Emotion recognition using bio-sensors: first steps towards an automatic system. In: André E, Dybkjær L, Minker W, Heisterkamp P (eds) Affective dialogue systems. Springer Berlin Heidelberg, New York, pp 36–48. doi:10.1007/978-3-540-24842-2_4
Jang E-H, Park B-J, Kim S-H, Chung M-A, Park M-S, Sohn J-H (2014) Emotion classification based on bio-signals emotion recognition using machine learning algorithms. In: Proceedings of 2014 international conference on information science, Electronics and Electrical Engineering (ISEEE), Sapporo, Japan, 26–28 April 2014, pp 104–109. doi:10.1109/InfoSEEE.2014.6946144
Jerritta S, Murugappan M, Nagarajan R, Wan K (2011) Physiological signals based human emotion recognition: a review. In: 2011 IEEE 7th international colloquium on signal processing and its applications (CSPA), Penang, Malaysia, 4–6 March 2011, pp 410–415. doi:10.1109/CSPA.2011.5759912
Kätsyri J, Förger K, Mäkäräinen M, Takala T (2015) A review of empirical evidence on different uncanny valley hypotheses: support for perceptual mismatch as one road to the valley of eeriness. Front Psychol 6:390. doi:10.3389/fpsyg.2015.00390
Kim J, André E (2008) Emotion recognition based on physiological changes in music listening. IEEE Trans Pattern Anal 30(12):2067–2083. doi:10.1109/TPAMI.2008.26
Kim KH, Bang SW, Kim SR (2004) Emotion recognition system using short-term monitoring of physiological signals. Med Biol Eng Comput 42(3):419–427. doi:10.1007/BF02344719
Kipp M (2012) ANVIL: a universal video research tool. In: Durand J, Gut U, Kristofferson G (eds) Handbook of corpus phonology. Oxford University Press, Oxford. doi:10.1093/oxfordhb/9780199571932.013.024
Kopp S, Krenn B, Marsella S, Marshall AN, Pelachaud C, Pirker H, Thórisson KR, Vilhjálmsson H (2006) Towards a common framework for multimodal generation: the behavior markup language. In: Gratch J, Young M, Aylett R, Ballin D, Olivier P (eds) Intelligent virtual agents. Springer International Publishing, Switzerland, pp 205–217. doi:10.1007/11821830_17
Liu B (2015) Sentiment analysis: mining opinions, sentiments, and emotions. Cambridge University Press, New York. doi:10.1017/CBO9781139084789
Marsella SC, Gratch J (2009) EMA: a process model of appraisal dynamics. Cogn Syst Res 10(1):70–90. doi:10.1016/j.cogsys.2008.03.005
Mathur MB, Reichling DB (2016) Navigating a social world with robot partners: a quantitative cartography of the Uncanny Valley. Cognition 146:22–32. doi:10.1016/j.cognition.2015.09.008
Nasoz F, Alvarez K, Lisetti CL, Finkelstein N (2003) Emotion recognition from physiological signals using wireless sensors for presence technologies. Cogn Technol Work 6(1):4–14. doi:10.1007/s10111-003-0143-x
Nass C, Lee KM (2000) Does computer-generated speech manifest personality? An experimental test of similarity-attraction. In: Proceedings of the SIGCHI conference on human factors in computing systems (CHI’00), The Hague, Netherlands, 1–6 April 2000, pp 329–336. doi:10.1145/332040.332452
Nass C, Yen C (2012) The man who lied to his laptop: what we can learn about ourselves from our machines. Penguin Group, New York
Niewiadomski R, Hyniewska SJ, Pelachaud C (2013) Computational models of expressive behaviors for a virtual agent. In: Gratch J, Marsella S (eds) Social emotions in nature and artifact. Oxford University Press, Oxford, pp 143–161. doi:10.1093/acprof:oso/9780195387643.003.0010
Paggio P, Navarretta C (2011) Head movements, facial expressions and feedback in danish first encounters interactions: a culture-specific analysis. In: Stephanidis C (ed) Universal access in human-computer interaction users diversity. Springer Berlin Heidelberg, New York, pp 583–590. doi:10.1007/978-3-642-21663-3_63
Pantic M, Bartlett MS (2007) Machine analysis of facial expressions. In: Delac K, Grgic M (eds) Face recognition. I-Tech Education and Publishing, Vienna, Austria, pp 377–416. doi:10.5772/4847
Pantic M, Valstar MF, Rademaker R, Maat L (2005) Web-based database for facial expression analysis. In: IEEE International conference on multimedia and expo (ICME), Amsterdam, The Netherlands, 6–8 July 2005, pp 317–321. doi:10.1109/ICME.2005.1521424
Pelachaud C (2009) Modelling multimodal expression of emotion in a virtual agent. Philos Trans R Soc B Biol Sci 364(1535):3539–3548. doi:10.1098/rstb.2009.0186
Petta P, Pelachaud C, Cowie R (eds) (2011) Emotion-oriented systems: the Humaine handbook. Springer, Berlin Heidelberg. doi:10.1007/978-3-642-15184-2
Picard RW, Vyzas E, Healey J (2001) Toward machine emotional intelligence: analysis of affective physiological state. IEEE Trans Pattern Anal 23(10):1175–1191. doi:10.1109/34.954607
Polzin TS, Waibel A (2000) Emotion-sensitive human-computer interfaces. In: International speech communication association (ISCA) tutorial and research workshop on speech and emotion. Newcastle, Northern Ireland, UK, pp 201–206
Ryan A, Cohn JF, Lucey S, Saragih J, Lucey P, De La Torre F, Rossi A (2009) Automated facial expression recognition system. In: 43rd annual international Carnahan conference on security technology, Zurich, Switzerland, 5–8 Oct 2009, pp 172–177. doi:10.1109/CCST.2009.5335546
Schröder M (2009) Expressive speech synthesis: past, present, and possible futures. In: Tao J, Tan T (eds) Affective information processing. Springer, London, pp 111–126. doi:10.1007/978-1-84800-306-4_7
Schuller B, Batliner A (2013) Computational paralinguistics: emotion, affect and personality in speech and language processing. Wiley, Chichester, UK. doi:10.1002/9781118706664
Van Santen J, Mishra T, Klabbers E (2008) Prosodic processing. In: Benesty J, Sondhi MM, Huang Y (eds) Springer handbook of speech processing. Springer, Berlin Heidelberg, pp 471–488. doi:10.1007/978-3-540-49127-9_23
Väyrynen E (2014) Emotion recognition from speech using prosodic features. Doctoral Dissertation, University of Oulu, Finland. http://urn.fi/urn:isbn:9789526204048
Ververidis D, Kotropoulos C (2006) Emotional speech recognition: resources, features and methods. Speech Commun 48(9):1162–1181. doi:10.1016/j.specom.2006.04.003
Wagner J, Lingenfelser F, Baur T, Damian I, Kistler F, André E (2013) The social signal interpretation (SSI) framework: multimodal signal processing and recognition in real-time. In: Proceedings of the 21st ACM international conference on Multimedia (MM’13), Barcelona, Spain, 21–25 Oct 2013, pp 831–834 doi:10.1145/2502081.2502223
Zhang C, Zhang Z (2010) A survey of recent advances in face detection. Microsoft TechReport MSR-TR-2010-66. http://research.microsoft.com/apps/pubs/default.aspx?id=132077
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
Copyright information
© 2016 Springer International Publishing Switzerland
About this chapter
Cite this chapter
McTear, M., Callejas, Z., Griol, D. (2016). Affective Conversational Interfaces. In: The Conversational Interface. Springer, Cham. https://doi.org/10.1007/978-3-319-32967-3_15
Download citation
DOI: https://doi.org/10.1007/978-3-319-32967-3_15
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-32965-9
Online ISBN: 978-3-319-32967-3
eBook Packages: EngineeringEngineering (R0)