Multimodal Corpora

From Models of Natural Interaction to Systems and Applications

  • Michael Kipp
  • Jean-Claude Martin
  • Patrizia Paggio
  • Dirk Heylen
Book MMCorp 2008

Part of the Lecture Notes in Computer Science book series (LNCS, volume 5509)

Also part of the Lecture Notes in Artificial Intelligence book sub series (LNAI, volume 5509)

Table of contents

  1. Front Matter
  2. Collecting and Distributing Multimodal Corpora

    1. R. J. J. H. van Son, Wieneke Wesseling, Eric Sanders, Henk van den Heuvel
      Pages 18-37
  3. Coding and Analyzing Multimodal Corpora

    1. Philippe Blache, Roxane Bertrand, Gaëlle Ferré
      Pages 38-53
    2. Jean-Marc Colletta, Ramona N. Kunene, Aurélie Venouil, Virginie Kaufmann, Jean-Pascal Simon
      Pages 54-72
    3. Isabella Poggi, Laura Vincze
      Pages 73-92
    4. Yosuke Matsusaka, Yasuhiro Katagiri, Masato Ishizaki, Mika Enomoto
      Pages 93-108
  4. Validating Multimodal Corpora

    1. Dennis Reidsma, Dirk Heylen, Rieks op den Akker
      Pages 122-137
  5. The Intercultural Dimension of Multimodal Corpora

    1. Matthias Rehm, Elisabeth André, Nikolaus Bee, Birgit Endrass, Michael Wissner, Yukiko Nakano et al.
      Pages 138-159
  6. Tools for Browsing, Coding and Exchanging Data in Multimodal Corpora

    1. Andrei Popescu-Belis, Jean Carletta, Jonathan Kilgour, Peter Poller
      Pages 189-206
    2. Thomas Schmidt, Susan Duncan, Oliver Ehmer, Jeffrey Hoyt, Michael Kipp, Dan Loehr et al.
      Pages 207-221
  7. Back Matter

About this book


Empirically based research in multimodality starts with the collection of video data relevant for a specific research question. This data must then be enriched by human coders, who annotate the corpus and who also explore the potential interconnections between modalities (e.g., face, gesture, speech, posture, etc.). The data must then be validated for consistency using quantitative measures -- systematic analysis of the observed multimodal behaviors is the central characteristic of multimodal corpora. For analysis and modeling tasks, exploratory qualitative and quantitative analysis tools are needed for browsing, viewing, extraction, and modeling. At the same time, the intercultural dimension must be taken into account, since multimodal interaction is always embedded in a social setting.

This state-of-the-art survey documents the scientific outcome of the International Workshop on "Multimodal Corpora: From Models of Natural Interaction to Systems and Applications", held in conjunction with the 6th International Conference for Language Resources and Evaluation (LREC), on May 27, 2008, in Marrakech, Morocco. It presents current research from fields as diverse as psychology, artificial intelligence, robotics, signal processing, computational linguistics, and human-computer interaction.

The 6 selected workshop contributions are complemented by 7 invited research articles, including contributions from major international multimodal corpus projects like AMI and SmartWeb. All papers underwent a special review process for this volume, resulting in significant revisions and extensions based on the experts' advice.


BAfH HCI artificial intelligence clustering emotion recognition gestures modeling multimodal behavior multimodal corpora multimodal interaction pattern recognition signal processing smart web corpora visualization

Editors and affiliations

  • Michael Kipp
    • 1
  • Jean-Claude Martin
    • 2
  • Patrizia Paggio
    • 3
  • Dirk Heylen
    • 4
  1. 1.Deutsches Forschungszentrum für künstliche Intelligenz (DFKI)SaarbrückenGermany
  2. 2.Laboratoire d’Informatique pour la Mécanique et les Sciences de l’Ingénieur (LIMSI-CNRS)Orsay CedexFrance
  3. 3.Faculty of Humanities, Centre for Language TechnologyUniversity of CopenhagenCopenhagenDenmark
  4. 4.Computer Science, Human Media InteractionUniversity of TwenteEnschedeThe Netherlands

Bibliographic information

  • DOI
  • Copyright Information Springer-Verlag Berlin Heidelberg 2009
  • Publisher Name Springer, Berlin, Heidelberg
  • eBook Packages Computer Science Computer Science (R0)
  • Print ISBN 978-3-642-04792-3
  • Online ISBN 978-3-642-04793-0
  • Series Print ISSN 0302-9743
  • Series Online ISSN 1611-3349
  • Buy this book on publisher's site
Industry Sectors
IT & Software