Abstract
In social interaction, people tend to mimic their conversational partners both when they agree and disagree. Research on this phenomenon is complex but not recent in theory, and related studies show that mimicry can enhance social relationships, increase affiliation and rapport. However, automatically recognising such a phenomenon is still in its early development. In this paper, we analyse mimicry in the speech domain and propose a novel method by using hand-crafted low-level acoustic descriptors and autoencoders (AEs). Specifically, for each conversation, two AEs are built, one for each speaker. After training, the acoustic features of one speaker are tested with the AE that is trained on the features of her counterpart. The proposed approach is evaluated on a database consisting of almost 400 subjects from 6 different cultures, recorded in-the-wild. By calculating the AE’s reconstruction errors of all speakers and analysing the errors at different times in their interactions, we show that, albeit to different degrees from culture to culture, mimicry arises in most interactions.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
References
Amiriparian, S., Freitag, M., Cummins, N., Schuller, B.: Sequence to sequence autoencoders for unsupervised representation learning from audio. In: Proceedings of the DCASE 2017 Workshop, Munich, Germany (2017)
Baltrušaitis, T., Robinson, P., Morency, L.P.: OpenFace: An open source facial behavior analysis toolkit. In: Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV), Lake Placid, NY, pp. 1–10 (2016)
Bilakhia, S., Petridis, S., Pantic, M.: Audiovisual detection of behavioural mimicry. In: Proceedings of Humaine Association Conference on Affective Computing and Intelligent Interaction (ACII), Geneva, Switzerland, pp. 123–128 (2013)
Burgoon, J.K., Hubbard, A.E.: Cross-cultural and intercultural applications of expectancy violations theory and interaction adaptation theory. In: Gudykunst, W.B. (ed.) Theorizing about Intercultural Communication, pp. 149–171. Sage Publications, Beverly Hills (2005)
Chartrand, T.L., Bargh, J.A.: The chameleon effect: the perception-behavior link and social interaction. J. Pers. Soc. Psychol. 76(6), 893–910 (1999)
Delaherche, E., Chetouani, M., Mahdhaoui, A., Saint-Georges, C., Viaux, S., Cohen, D.: Interpersonal synchrony: a survey of evaluation methods across disciplines. IEEE Trans. Affect. Comput. 3(3), 349–365 (2012)
Ekman, P.: Universals and cultural differences in facial expressions of emotion. In: Nebraska Symposium on Motivation. University of Nebraska Press (1971)
Ekman, P., Friesen, W.V.: Unmasking the Face: A Guide to Recognizing Emotions from Facial Clues. Ishk, Ujjain (2003)
Eyben, F.: Real-Time Speech and Music Classification by Large Audio Feature Space Extraction. Springer, Switzerland (2016). https://doi.org/10.1007/978-3-319-27299-3
Eyben, F., Weninger, F., Groß, F., Schuller, B.: Recent developments in openSMILE, the Munich open-source multimedia feature extractor. In: Proceedings of the 21st ACM International Conference on Multimedia (MM), Barcelona, Spain, pp. 835–838 (2013)
Friesen, E., Ekman, P.: Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, Palo Alto (1978)
Gueguen, N., Jacob, C., Martin, A.: Mimicry in social interaction: its effect on human judgment and behavior. Eur. J. Soc. Sci. 8(2), 253–259 (2009)
Hess, U., Fischer, A.: Emotional mimicry as social regulation. Pers. Soc. Psychol. Rev. 17(2), 142–157 (2013)
Hess, U., Fischer, A.: Emotional mimicry: why and when we mimic emotions. Soc. Pers. Psychol. Compass 8(2), 45–57 (2014)
Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Advances in Neural Information Processing Systems (NIPS), Lake Tahoe, NV, pp. 3111–3119 (2013)
Parrill, F., Kimbara, I.: Seeing and hearing double: the influence of mimicry in speech and gesture on observers. J. Nonverbal Behav. 30(4), 157 (2006)
Ringeval, F., et al.: AVEC 2018 workshop and challenge: Bipolar disorder and cross-cultural affect recognition. In: Proceedings of the 8th Annual Workshop on Audio/Visual Emotion Challenge, Seoul, Korea (2018, to appear)
Ringeval, F., et al.: AVEC 2017: Real-life depression, and affect recognition workshop and challenge. In: Proc. of the 7th Annual Workshop on Audio/Visual Emotion Challenge, Mountain View, CA, pp. 3–9 (2017)
Schmitt, M., Ringeval, F., Schuller, B.: At the border of acoustics and linguistics: bag-of-audio-words for the recognition of emotions in speech. In: Proceedings of INTERSPEECH, San Francisco, CA, pp. 495–499 (2016)
Schuller, B., et al.: The INTERSPEECH 2013 computational paralinguistics challenge: Social signals, conflict, emotion, autism. In: Proceedings of INTERSPEECH, Lyon, France, pp. 148–152 (2013)
Scissors, L.E., Gill, A.J., Gergle, D.: Linguistic mimicry and trust in text-based CMC. In: Proceedings of the ACM Conference on Computer Supported Cooperative Work, San Diego, CA, pp. 277–280 (2008)
Stivers, T., et al.: Universals and cultural variation in turn-taking in conversation. Proc. Nat. Acad. Sci. U.S.A 106(26), 10587–10592 (2009)
Sun, X., Nijholt, A., Truong, K.P., Pantic, M.: Automatic visual mimicry expression analysis in interpersonal interaction. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Colorado Springs, CO, pp. 40–46 (2011)
Swaab, R.I., Maddux, W.W., Sinaceur, M.: Early words that work: When and how virtual linguistic mimicry facilitates negotiation outcomes. J. Exper. Soc. Psychol. 47(3), 616–621 (2011)
Acknowledgments
The research leading to these results has received funding from the European Union’s Horizon 2020 Programme under GA No. 645094 (Innovation Action SEWA) and through the EFPIA Innovative Medicines Initiative under GA No. 115902 (RADAR-CNS).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Han, J., Schmitt, M., Schuller, B. (2018). You Sound Like Your Counterpart: Interpersonal Speech Analysis. In: Karpov, A., Jokisch, O., Potapova, R. (eds) Speech and Computer. SPECOM 2018. Lecture Notes in Computer Science(), vol 11096. Springer, Cham. https://doi.org/10.1007/978-3-319-99579-3_20
Download citation
DOI: https://doi.org/10.1007/978-3-319-99579-3_20
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-99578-6
Online ISBN: 978-3-319-99579-3
eBook Packages: Computer ScienceComputer Science (R0)