Abstract
A new experimental method based on the dual task paradigm is used to evaluate speech intelligibility of an embodied conversational agent (ECA). The experiment consists of the manipulation of auditory-visual (AV) versus auditory-only (A) presentation of speech. In the dual task, participants perform two tasks concurrently. The secondary task is sensitive to cognitive processing demands of the primary task. In the primary task participants either shadowed words or named the superordinate categories to which words belonged, as the word items were spoken by the ECA under A or AV conditions. Reaction time (RT) on the secondary task–swatting a fly on the ECA face–was affected by the difficulty of the concurrent task. The secondary RT was affected by modality of presentation of the primary task. Using a relatively primitive ECA, RT on the secondary task was significantly slower when shadowing occurred in AV versus A conditions. The benefits of this evaluation system, that returns quantitative behavioural data and self-report ratings, are discussed.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Ibister, K., Doyle, P.: Design and evaluation of embodied conversational agents: a proposed taxonomy. In: Proc 1st Intl Joint Conf Autonomous Agents and Multi-Agent System, AAMAS 2002, Bologna, Italy (2002)
Catrambone, R., Stasko, J., Xiao, J.: ECA as user interface paradigm: experimental findings within a framework for research. In: Ruttkay, Z., Pelachaud, C. (eds.) From Brows to Trust: Evaluating Embodied Conversational Agents, pp. 239–267. Springer, The Netherlands (2005)
Buisine, S., Abrillian, S., Martin, J.-C.: Evaluation of multimodal behaviour of embodied agents. In: Ruttkay, Z., Pelachaud, C. (eds.) From Brows to Trust: Evaluating Embodied Conversational Agents, pp. 217–238. Springer, The Netherlands (2005)
Koller, A., Striegnitz, K., Byron, D., Cassell, J., Dale, R., Dalzel-Job, S., Oberlander, J., Moore, J.: Validating the web-based evaluation of NLG systems. In: Proc ACL-IJCNLP, Singapore (2009)
Ruttkay, Z., Pelachaud, C. (eds.): From brows to trust: evaluating embodied conversational agents. Kluwer Academic Publishers, Dordrecht (2005)
Sharp, H., Rogers, Y., Preece, J.: Interaction design: Beyond human-computer interaction, 2nd edn. John Wiley & Sons, Ltd., Chichester (2007)
Stevens, C., Lees, N., Vonwiller, J., Burnham, D.: On-line experimental methods to evaluate text-to-speech (TTS) synthesis: Effects of voice gender and signal quality on intelligibility, naturalness and preference. Computer Speech & Language 19, 129–146 (2005)
Kopp, S., Gesellensetter, L., Krämer, N.C., Wachsmuth, I.: A conversational agent as museum guide – design and evaluation of a real-world application. In: Panayiotopoulos, T., Gratch, J., Aylett, R.S., Ballin, D., Olivier, P., Rist, T. (eds.) IVA 2005. LNCS (LNAI), vol. 3661, pp. 329–343. Springer, Heidelberg (2005)
Bailly, G., Raidt, S., Elisei, F.: Gaze, conversational agents and face-to-face communication. Speech Comm. 52, 598–612 (2010)
Badin, P., Tarabalka, Y., Elisei, F., Bailly, G.: Can you ‘read’ tongue movements? Evaluation of the contribution of tongue display to speech understanding. Speech Comm. 52, 493–503 (2010)
Karatekin, C., Couperus, J.W., Marcus, D.J.: Attention allocation in the dual-task paradigm as measured through behavioral and psychophysiological responses. Psychophysiol. 41, 175–185 (2004)
Pashler, H., Johnston, J.C.: Attentional limitations in dual-task performance. In: Pashler, H. (ed.) Attention, pp. 155–189. Psychology Press, East Sussex (1998)
Johnston, W.A., Heinz, S.P.: Flexibility and capacity demands of attention. J. Experimental Psychol.: General 107, 420–435 (1978)
Wickens, C.D.: Multiple resources and performance prediction. Theoretical Issues in Ergonomic Science 3, 159–177 (2002)
Fisk, A.D., Derrick, W.L., Schneider, W.: A methodological assessment and evaluation of dual-task paradigms. Current Psychological Research & Reviews 5, 315–327 (1986)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2011 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Stevens, C.J., Gibert, G., Leung, Y., Zhang, Z. (2011). A Flexible Dual Task Paradigm for Evaluating an Embodied Conversational Agent: Modality Effects and Reaction Time as an Index of Cognitive Load. In: Vilhjálmsson, H.H., Kopp, S., Marsella, S., Thórisson, K.R. (eds) Intelligent Virtual Agents. IVA 2011. Lecture Notes in Computer Science(), vol 6895. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-23974-8_36
Download citation
DOI: https://doi.org/10.1007/978-3-642-23974-8_36
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-23973-1
Online ISBN: 978-3-642-23974-8
eBook Packages: Computer ScienceComputer Science (R0)