© 2011

Interactive Multi-modal Question-Answering

  • Antal van den Bosch
  • Gosse Bouma


  • A unique peek into the kitchen of a large multidisciplinary HCI project

  • New perspectives on question-answering systems

  • Unique new developments in multi-model answer generation, fusion, and paraphrasing

  • New insights in speech recognition technology Theoretical advancements in dialogue management


Table of contents

  1. Front Matter
    Pages i-xii
  2. Introduction to the IMIX Programme

    1. Front Matter
      Pages 1-1
    2. Antal van den Bosch, Gosse Bouma
      Pages 3-9
    3. Dennis Hofs, Boris van Schooten, Rieks op den Akker
      Pages 11-21
  3. Interaction Management

    1. Front Matter
      Pages 23-23
    2. Simon Keizer, Harry Bunt, Volha Petukhova
      Pages 57-86
  4. Fusing Text, Speech, and Images

    1. Front Matter
      Pages 87-87
    2. Charlotte van Hooijdonk, Wauter Bosma, Emiel Krahmer, Alfons Maes, Mariët Theune
      Pages 89-115
    3. Wauter Bosma, Erwin Marsi, Emiel Krahmer, Mariët Theune
      Pages 117-145
  5. Text Analysis for Question Answering

    1. Front Matter
      Pages 147-147
    2. Lonneke van der Plas, Jörg Tiedemann, Ismail Fahmi
      Pages 149-170
    3. Gosse Bouma, Ismail Fahmi, Jori Mur
      Pages 171-197
    4. Sander Canisius, Antal van den Bosch, Walter Daelemans
      Pages 199-221
    5. Erik Tjong Kim Sang, Katja Hofmann, Maarten de Rijke
      Pages 223-245
  6. Epilogue

    1. Front Matter
      Pages 269-269
    2. Eduard Hovy, Jon Oberlander, Norbert Reithinger
      Pages 271-279

About this book


This book is the result of a group of researchers from different disciplines asking themselves one question: what does it take to develop a computer interface that listens, talks, and can answer questions in a domain? First, obviously, it takes specialized modules for speech recognition and synthesis, human interaction management (dialogue, input fusion, andmultimodal output fusion), basic question understanding, and answer finding. While all modules are researched as independent subfields, this book describes the development of state-of-the-art modules and their integration into a single, working application capable of answering medical (encyclopedic) questions such as "How long is a person with measles contagious?" or "How can I prevent RSI?".

The contributions in this book, which grew out of the IMIX project funded by the Netherlands Organisation for Scientific Research, document the development of this system, but also address more general issues in natural language processing, such as the development of multidimensional dialogue systems, the acquisition of taxonomic knowledge from text, answer fusion, sequence processing for domain-specific entity recognition, and syntactic parsing for question answering. Together, they offer an overview of the most important findings and lessons learned in the scope of the IMIX project, making the book of interest to both academic and commercial developers of human-machine interaction systems in Dutch or any other language.

Highlights include: integrating multi-modal input fusion in dialogue management (Van Schooten and Op den Akker), state-of-the-art approaches to the extraction of term variants (Van der Plas, Tiedemann, and Fahmi; Tjong Kim Sang, Hofmann, and De Rijke), and multi-modal answer fusion (two chapters by Van Hooijdonk, Bosma, Krahmer, Maes, Theune, and Marsi).

Watch the IMIX movie at .

Like IBM's Watson, the IMIX system described in the book gives naturally phrased responses to naturally posed questions. Where Watson can only generate synthetic speech, the IMIX system also recognizes speech. On the other hand, Watson is able to win a television quiz, while the IMIX system is domain-specific, answering only to medical questions.

"The Netherlands has always been one of the leaders in the general field of Human Language Technology, and IMIX is no exception. It was a very ambitious program, with a remarkably successful performance leading to interesting results. The teams covered a remarkable amount of territory in the general sphere of multimodal  question answering and information delivery, question answering, information extraction and component technologies."

Eduard Hovy, USC, USA, Jon Oberlander, University of Edinburgh, Scotland, and Norbert Reithinger, DFKI, Germany 


Dialogue management Multimodal interaction Question answering Speech technology System integration

Editors and affiliations

  • Antal van den Bosch
    • 1
  • Gosse Bouma
    • 2
  1. 1.Fac. HumanitiesTilburg UniversityTilburgNetherlands
  2. 2., Information ScienceUniversity of GroningenNL-9700 AS GroningenNetherlands

Bibliographic information

Industry Sectors
IT & Software


From the reviews:

“Researchers in broad disciplines … usually conduct their research by breaking the general problem down into many small problems. … It is rare for many researchers to coordinate their efforts and demonstrate what progress has been made to solve the general problem. This book documents one such effort. … The book is a collection of chapters written by various researchers in the IMIX project. … Anyone who wants to know about the state of the art in question answering will be interested in this book.” (D. L. Chester, ACM Computing Reviews, August, 2011)