The MultiLis Corpus – Dealing with Individual Differences in Nonverbal Listening Behavior

  • Iwan de Kok
  • Dirk Heylen
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6456)


Computational models that attempt to predict when a virtual human should backchannel are often based on the analysis of recordings of face-to-face conversations between humans. Building a model based on a corpus brings with it the problem that people differ in the way they behave. The data provides examples of responses of a single person in a particular context but in the same context another person might not have provided a response. Vice versa, the corpus will contain contexts in which the particular listener recorded did not produce a backchannel response, where another person would have responded. Listeners can differ in the amount, the timing and the type of backchannels they provide to the speaker, because of individual differences - related to personality, gender, or culture, for instance. To gain more insight in this variation we have collected data in which we record the behaviors of three listeners interacting with one speaker. All listeners think they are having a one-on-one conversation with the speaker, while the speaker actually only sees one of the listeners. The context, in this case the speaker’s actions, is for all three listeners the same and they respond to it individually. This way we have created data on cases in which different persons show similar behaviors and cases in which they behave differently. With the recordings of this data collection study we can start building our model of backchannel behavior for virtual humans that takes into account similarities and differences between persons.


Multimodal corpus listeners task-oriented 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Brugman, H., Russel, A.: Annotating multimedia/multi-modal resources with ELAN. In: Proceedings of the Fourth International Conference on Language Resources and Evaluation, Citeseer, pp. 2065–2068 (2004)Google Scholar
  2. 2.
    Cathcart, N., Carletta, J., Klein, E.: A shallow model of backchannel continuers in spoken dialogue. In: European ACL, pp. 51–58 (2003)Google Scholar
  3. 3.
    Gratch, J., Wang, N., Gerten, J., Fast, E., Duffy, R.: Creating rapport with virtual agents. In: Pelachaud, C., Martin, J.-C., André, E., Chollet, G., Karpouzis, K., Pelé, D. (eds.) IVA 2007. LNCS (LNAI), vol. 4722, pp. 125–138. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  4. 4.
    Huang, L., Morency, L.-P., Gratch, J.: Parasocial Consensus Sampling: Combining Multiple Perspectives to Learn Virtual Human Behavior. In: Proceedings of Autonomous Agents and Multi-Agent Systems, Toronto, Canada (2010)Google Scholar
  5. 5.
    Huijbregts, M.: Segmentation, Diarization and Speech Transcription: Surprise Data Unraveled. Phd thesis, University of Twente (2008)Google Scholar
  6. 6.
    John, O.P., Naumann, L.P., Soto, C.J.: Paradigm shift to the integrative Big-Five trait taxonomy: History, measurement, and conceptual issues, 3rd edn., ch. 4, pp. 114–158. Guilford Press, New York (2008)Google Scholar
  7. 7.
    Morency, L.P., de Kok, I., Gratch, J.: A probabilistic multimodal approach for predicting listener backchannels. Autonomous Agents and Multi-Agent Systems 20(1), 70–84 (2010)CrossRefGoogle Scholar
  8. 8.
    Noguchi, H., Den, Y.: Prosody-based detection of the context of backchannel responses. In: Fifth International Conference on Spoken Language Processing (1998)Google Scholar
  9. 9.
    Terry, P.C., Lane, A.M., Fogarty, G.J.: Construct validity of the Profile of Mood States-Adolescents for use with adults. Psychology of Sport and Exercise 4(2), 125–139 (2003)CrossRefGoogle Scholar
  10. 10.
    Ward, N., Tsukahara, W.: Prosodic features which cue back-channel responses in English and Japanese. Journal of Pragmatics 32(8), 1177–1207 (2000)CrossRefGoogle Scholar
  11. 11.
    Watson, D., Clark, L.A.: The PANAS-X (1994)Google Scholar
  12. 12.
    White, S.: Backchannels across cultures: A study of Americans and Japanese. Language in Society 18(1), 59–76 (1989)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Iwan de Kok
    • 1
  • Dirk Heylen
    • 1
  1. 1.Human Media InteractionUniversity of TwenteThe Netherlands

Personalised recommendations