Abstract
The current study focuses on multilingual speech emotion recognition using realistic emotional speech extracted from English, Italian, and Spanish films. Two novel methods are proposed, which exploit language information and emotion information. In the first method, features specific to the three languages are concatenated with emotion-specific features and applied using a common extremely randomized trees (ERT) classifier to recognize five emotions. In the second method, a stacked generalization ensemble (SGE) with two ERTs for language and emotion are employed. On top, another ERT is used as a meta-classifier for the final recognition of five emotions. Using the feature fusion-based method, a 73.3% unweighted average recall (UAR) was achieved. This result is very promising and superior to the UAR obtained by human evaluation (71.8% for Italian instances). When using the SGE-based method, a 69.2% UAR was achieved, which is closely comparable to the human evaluation results.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Busso, C., Bulut, M., Narayanan, S.: Toward effective automatic recognition systems of emotion in speech. In: Gratch, J., Marsella, S. (eds.) Social Emotions in Nature and Artifact: Emotions in Human and Human-Computer Interaction, pp. 110–127. Oxford University Press, New York (2013)
Geurts, P., Ernst, D., Wehenkel, L.: Extremely randomized trees. Mach. Learn. 63(1), 3–42 (2006)
Dehak, N., Kenny, P.J., Dehak, R., Dumouchel, P., Ouellet, P.: Front-end factor analysis for speaker verification. IEEE Trans. Audio Speech Lang. Process. 19(4), 788–798 (2011)
P-Cabaleiro, E., Costantini, G., Batliner, A., Baird, A., Schuller, B.: Categorical vs dimensional perception of Italian emotional speech. In: Proceedings of Interspeech, pp. 3638–3642 (2018)
Schuller, B., Rigoll, G., Lang, M.: Hidden Markov model-based speech emotion recognition. In: Proceedings of the IEEE ICASSP, vol. I, pp. 401–404 (2003)
Nicholson, J., Takahashi, K., Nakatsu, R.: Emotion recognition in speech using neural networks. Neural Comput. Appl. 9(4), 290–296 (2000)
Pan, Y., Shen, P., Shen, L.: Speech emotion recognition using support vector machine. Int. J. Smart Home 6(2), 101–108 (2012)
Stuhlsatz, A., Meyer, C., Eyben, F., Zielke1, T., Meier, G., Schuller, B.: Deep neural networks for acoustic emotion recognition: raising the benchmarks. In: Proceedings of ICASSP, pp. 5688–5691 (2011)
Han, K., Yu, D., Tashev, I.: Speech emotion recognition using deep neural network and extreme learning machine. In: Proceedings of Interspeech, pp. 2023–2027 (2014)
Sahidullah, M., Saha, G.: Design, analysis and experimental evaluation of block based transformation in MFCC computation for speaker recognition. Speech Commun. 54(4), 543–565 (2012)
Schuller, B.W., et al.: The INTERSPEECH 2016 computational paralinguistics challenge: deception, sincerity & native language. In: Proceedings of Interspeech, 2001–2005 (2016)
Feraru, S.M., Schuller, D., Schuller, B.: Cross-language acoustic emotion recognition: an overview and some tendencies. In: International Conference on Affective Computing and Intelligent Interaction (ACII), 125–131 (2015)
Li, X., Akagi, M.: Improving multilingual speech emotion recognition by combining acoustic features in a three-layer model. Speech Commun. 110, 1–12 (2019)
Polzehl, T., Schmitt, A., Metze, F.: Approaching multi-lingual emotion recognition from speech-on language dependency of acoustic prosodic features for anger detection. In: Proceedings of Speech Prosody (2010)
Bhaykar, M., Yadav, J., Rao, K.S.: Speaker dependent, speaker independent and cross language emotion recognition from speech using GMM and HMM. In: 2013 National Conference on Communications (NCC), pp. 1–5. IEEE (2013)
Eyben, F., Batliner, A., Schuller, B., Seppi, D., Steidl, S.: Crosscorpus classification of realistic emotions - some pilot experiments. In: Proceedings of the Third International Workshop on EMOTION (Satellite of LREC) (2010)
Heracleous, P., Mohammad, Y., Yoneyama, A.: Deep convolutional neural networks for feature extraction in speech emotion recognition. In: Kurosu, M. (ed.) HCII 2019. LNCS, vol. 11567, pp. 117–132. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-22643-5_9
Heracleous, P., Yoneyama, A.: A comprehensive study on bilingual and multilingual speech emotion recognition using a two-pass classification scheme (2019). https://doi.org/10.1371/journal.pone.0220386
Ververidis, D., Kotropoulos, C.: Emotional speech recognition: resources, features, and methods. Speech Commun. 48(9), 1162–1181 (2006)
Bielefeld, B.: Language identification using shifted delta cepstrum. In: Fourteenth Annual Speech Research Symposium (1994)
Ho, T.K.: Random decision forests. In: Proceedings of the 3rd International Conference on Document Analysis and Recognition, pp. 278–282 (1995)
Sagha, H., Matejka, P., Gavryukova, M., Povolný, F., Marchi, E., Schuller, B.W.: Enhancing multilingual recognition of emotion in speech by language identification. In: Proceedings of Interspeech, pp. 2949–2953 (2016)
Fukunaga, K.: Introduction to Statistical Pattern Recognition, 2nd edn. Academic Press, New York (1990). Ch. 10
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Heracleous, P., Mohammad, Y., Yoneyama, A. (2020). Integrating Language and Emotion Features for Multilingual Speech Emotion Recognition. In: Kurosu, M. (eds) Human-Computer Interaction. Multimodal and Natural Interaction. HCII 2020. Lecture Notes in Computer Science(), vol 12182. Springer, Cham. https://doi.org/10.1007/978-3-030-49062-1_12
Download citation
DOI: https://doi.org/10.1007/978-3-030-49062-1_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-49061-4
Online ISBN: 978-3-030-49062-1
eBook Packages: Computer ScienceComputer Science (R0)