Software Engineering for Multimodal Interactive Systems

Abstract

The power and versatility of multimodal interfaces results in an increased complexity of the code to be developed. To address this problem of software development, multimodal interfaces therefore make necessary the definition of software development tools that satisfy specific requirements such as the fusion of data from different interaction modalities and the management of multiple processes including support for synchronization and race conditions between distinct interaction modalities. This chapter is a reflection on current software design methods and tools for multimodal interfaces. First we present the PAC-Amodeus software architectural model as a generic conceptual solution for addressing the challenges of data fusion and concurrent processing of data of multimodal interfaces. We then focus on software tools for developing multimodal interfaces. The existing tools dedicated to multimodal interfaces are currently few and limited in scope. Either they address a specific technical problem including the fusion mechanism, the composition of several devices and mutual disambiguation, or they are dedicated to specific interaction modalities such as gesture recognition, speech recognition or the combined usage of speech and gesture. Two platforms, ICARE and OpenInterface are more general and are able to handle various interaction modalities. We describe in detail these two platforms and illustrate them by considering developed multimodal interfaces.