Conversational Multimedia Interaction

Abstract

More effective, efficient and natural human computer or computer mediated human-human interaction will require both automated understanding and generation of multimedia. Fluent conversational interaction demands explicit models of the user, discourse, task and context. It will also require a richer understanding of media (i.e., text, audio, video), both in its use in the interface to support interaction with the user as well as its use in access to content by the user during a session. Multimedia dialogue prototypes have been developed in several application domains including CUBRICON (for a mission planning domain) (Neal and Shapiro, 1991), XTRA (tax-form preparation) (Wahlster, 1991), AIMI (air mission planning) (Burger and Marshall, 1993), and AlFresco (art history information exploration) (Stock et al., 1993). Typically, these systems parse mixed (typically asynchronous) multimedia input and generate coordinated multimedia output. They also attempt to maintain coherency, cohesion, and consistency across both multimedia input and output. For example, these systems often support integrated language and deixis for both input and output. They extend research in discourse and user modeling (Kobsa and Wahlster, 1989) by incorporating representations of media to enable media (cross) reference and reuse over the course of a session with a user. These enhanced representations support the exploitation of user perceptual abilities and media preferences as well as the resolution of multimedia references (e.g. “Send this plane there” articulated with synchronous gestures on a map).