Patterns of Synchronization of Non-verbal Cues and Speech in ECAs: Towards a More “Natural” Conversational Agent
This paper presents an analysis of the verbal and non-verbal cues of Conversational Agents, with a special focus on REA and GRETA, in order to allow further research aimed at correcting some traits of their performance still considered unnatural by their final users. Despite the striking performance of new generation ECA, some important features make these conversational agents unreliable to the users, who usually prefer interacting with a classical computer for information retrieval. The users’ preference can be due to several factors, such as the quality of speech synthesis, or the inevitable unnaturalness of the graphics animating the avatar. Apart from the unavoidable traits that can render ECAs unnatural to the ultimate users, instances of poor synchronization between verbal and non-verbal behaviour may contribute to unfavourable results. An instance of synchronization patterns between non-verbal cues and speech is here analysed and re-applied to the basic architecture of an ECA in order to improve the ECA’s verbal and non-verbal synchronization. A proposal for future inquiry aimed at creating alternative model for the ultimate Mp4 output is also proposed, for further development in this field.
KeywordsEmbodied Conversational Agents Prosody Non-verbal Communication Expressions Gesture-Speech Synchronization
Unable to display preview. Download preview PDF.
- 2.Cassell, J., Stocky, T., Bickmore, T., Gao, Y., Nakano, Y., Ryokai, K., Tversky, D., Vaucelle, C., Vilhjálmsson, H.: MACK: Media lab Autonomous Conversational Kiosk. In: Proceedings of IMAGINA 2002, Monte Carlo, January 12-15 (2002), http://www.media.mit.edu/gnl/pubs/imagina02.pdf
- 3.Cassell, J.: Trading spaces: Gesture Morphology and Semantics in Humans and Virtual Humans. In: Second ISGS Conference “Interacting bodies”. École normale supérieure Lettres et Sciences humaines Lyon - France, June 15-18 (2005)Google Scholar
- 4.Cassell, J., Bickmore, T., Billinghurst, M., Campbell, L., Chang, K., Vilhjálmsson, Yan, H.: Embodiment in Conversational Interfaces: Rea. In: Proceedings of the CHI 1999 Conference, Pittsburgh, PA, pp. 520–527 (1999)Google Scholar
- 6.Cassell, J., Nakano, Y., Bickmore, T., Sidner, C., Rich, C.: Annotating and Generating Posture from Discourse Structure in Embodied Conversational Agents. In: Workshop on Representing, Annotating, and Evaluating Non-Verbal and Verbal Communicative Acts to Achieve Contextual Embodied Agents, Autonomous Agents 2001 Conference, Montreal, Quebec, May 29 (2001), http://www.ccs.neu.edu/home/bickmore/publications/agents01.pdf
- 10.McNeill, D.: Hand and Mind. What Gestures Reveal about Thought. University of Chicago Press, Chicago (1992)Google Scholar
- 11.Eibl-Eibesfeldt, I.: Similarities and differences between cultures in expressive movements. In: Hinde, A. (ed.) Non-verbal Communication, pp. 297–312. Cambridge University Press, Cambridge (1972)Google Scholar
- 12.Kita, S., van Gijn, I., van der Hulst, H.: The non-linguistic status of the Symmetry Condition in signed languages: Evidence from a comparison from signs and speech-accompanying representational gestures (in progress)Google Scholar
- 13.Rossini, N.: Sociolinguistics in gesture. How about the Mano a Borsa? In: Intercultural Communication Studies, XIII, 3, pp. 144–154; Proceedings of the 9th International Conference on Cross-Cultural Communication (2004)Google Scholar
- 14.Rossini, N.: Gesture and its cognitive origin: Why do we gesture? Experiments on hearing and deaf people. Università di Pavia Ph.D. thesis (2004)Google Scholar
- 15.Thies, A.: First the hand, then the word: On gestural displacement in non-native English speech. Universität Bielefeld Ph.D. Thesis (2003)Google Scholar
- 16.Gibbon, D.: Modelling gesture as speech: a linguistic approach. In: Proceedings of GESPIN 2009, Conference on Gestures and Speech in Interaction, Poznan (September 2009) (to appear)Google Scholar
- 17.Rossini, N.: Il gesto. Gestualità e tratti non verbali in interazioni diadiche. Bologna, Pitagora (2009)Google Scholar