The Role of Discourse Structure and Response Time in Multimodal Communication

Abstract

In an ongoing project on multimodal communication in humans and agents [1], we investigate the interaction between two linguistic modalities (prosody and dialog structure) and two non-linguistic modalities (eye gaze and facial expressions). The goal is to gain a better understanding of the use of communicative channels in discourse and can subsequently aid the development of more effective animated conversational agents. We studied conversations between humans involved in the Map Task scenario whereby an Instruction Giver (IG) navigates an Instruction Follower (IF) from a starting point to an end point on a map [2].