Multimodal Signals: Cognitive and Algorithmic Issues

1. Front Matter

   PDF

2. Interactive and Unsupervised Multimodal Systems

   1. Multimodal Human Machine Interactions in Virtual and Augmented Reality

      • Gérard Chollet, Anna Esposito, Annie Gentes, Patrick Horain, Walid Karam, Zhenbo Li et al.

      Pages 1-23

   2. Speech through the Ear, the Eye, the Mouth and the Hand

      • Marion Dohen

      Pages 24-39

   3. Multimodality Issues in Conversation Analysis of Greek TV Interviews

      • Maria Koutsombogera, Harris Papageorgiou

      Pages 40-46

   4. Representing Communicative Function and Behavior in Multimodal Communication

      • Hannes Högni Vilhjálmsson

      Pages 47-59

   5. Using the iCat as Avatar in Remote Meetings

      • Dirk Heylen, Mannes Poel, Anton Nijholt

      Pages 60-66

   6. Using Context to Disambiguate Communicative Signals

      • Mark ter Maat, Dirk Heylen

      Pages 67-74

   7. Modeling Aspects of Multimodal Lithuanian Human - Machine Interface

      • Rytis Maskeliunas

      Pages 75-82

   8. Using a Signing Avatar as a Sign Language Research Tool

      • Delroy Nelson

      Pages 83-93

   9. Data Fusion at Different Levels

      • Marcos Faundez-Zanuy

      Pages 94-103

   10. Voice Technology Applied for Building a Prototype Smart Room

       • Josef Chaloupka, Jan Nouza, Jindrich Zdansky, Petr Cerva, Jan Silovsky, Martin Kroul

       Pages 104-111

   11. Towards Facial Gestures Generation by Speech Signal Analysis Using HUGE Architecture

       • Goranka Zoric, Karlo Smid, Igor S. Pandzic

       Pages 112-120

   12. Multi-modal Speech Processing Methods: An Overview and Future Research Directions Using a MATLAB Based Audio-Visual Toolbox

       • Andrew Abel, Amir Hussain

       Pages 121-129

   13. From Extensity to Protensity in CAS: Adding Sounds to Icons

       • Alina E. Lascu, Alexandru V. Georgescu

       Pages 130-136

   14. Statistical Modeling of Interpersonal Distance with Range Imaging Data

       • René Hempel, Patrick Westfeld

       Pages 137-144

3. Verbal and Nonverbal Communication Signals

   1. How the Brain Processes Language in Different Modalities

      • Bencie Woll

      Pages 145-163

   2. From Speech and Gestures to Dialogue Acts

      • Maciej Karpinski

      Pages 164-169

   3. The Language of Interjections

      • Isabella Poggi

      Pages 170-186

   4. Gesture and Gaze in Persuasive Political Discourse

      • Laura Vincze

      Pages 187-196

   5. Content in Embedded Sentences

      • Uli Sauerland, Mathias Schenner

      Pages 197-207

This book constitutes the thoroughly refereed post-conference proceedings of the COST Action 2102 and euCognition supported international school on Multimodal Signals: 'Cognitive and Algorithmic Issues' held in Vietri sul Mare, Italy, in April 2008. The 34 revised full papers presented were carefully reviewed and selected from participants’ contributions and invited lectures given at the workshop. The volume is organized in two parts; the first on Interactive and Unsupervised Multimodal Systems contains 14 papers. The papers deal with the theoretical and computational issue of defining algorithms, programming languages, and determinist models to recognize and synthesize multimodal signals. These are facial and vocal expressions of emotions, tones of voice, gestures, eye contact, spatial arrangements, patterns of touch, expressive movements, writing patterns, and cultural differences, in anticipation of the implementation of intelligent avatars and interactive dialogue systems that could be exploited to improve user access to future telecommunication services. The second part of the volume, on Verbal and Nonverbal Communication Signals, presents 20 original studies devoted to the modeling of timing synchronisation between speech production, gestures, facial and head movements in human communicative expressions and on their mutual contribution for an effective communication.

• algorithms
• animation systems
• architecture
• audiovisual processing
• avatar
• cognition
• communication
• computer-assisted surgery
• conversation analysis
• dialects
• emotion recognition
• gesture
• modeling
• programming language
• speech processing

• Department of Psychology, Second University of Naples, and IIASS, Vietri sul Mare (SA), Italy

  Anna Esposito

• Department of Computing Science & Mathematics, University of Stirling, Stirling, Scotland, UK

  Amir Hussain

• Dipartimento di Fisica “E.R. Caianiello”, Università degli Studi di Salerno, Italy and IIASS, Baronissi (SA), Italy

  Maria Marinaro

• Dip. di Ingegneria dell’ Informazione, Seconda Università di Napoli, Aversa (CE), Italy

  Raffaele Martone

Policies and ethics