Representing Communicative Function and Behavior in Multimodal Communication

  • Hannes Högni Vilhjálmsson
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5398)


In this paper I discuss how communicative behavior can be represented at two levels of abstraction, namely the higher level of communicative intent or function, which does not make any claims about the surface form of the behavior, and the lower level of physical behavior description, which in essence instantiates intent as a particular multimodal realization. I briefly outline the proposed SAIBA framework for multimodal generation of communicative behavior, which is an international research platform that fosters the exchange of components between different systems. The SAIBA framework currently contains a first draft of the lower level Behavior Markup Language (BML) and is starting work on the higher level Function Markup Language (FML). I also briefly explain the usefulness of this distinction by using examples of several implemented systems that each draws different strengths from it. These systems range from autonomous conversational agents to computer mediated communication.


Communicative behavior communicative function multimodal communication embodied conversational agents mediated communication 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Argyle, M., Cook, M.: Gaze and mutual gaze. Cambridge University Press, Cambridge (1976)Google Scholar
  2. 2.
    Argyle, M., Ingham, R., Alkema, F., et al.: The Different Functions of Gaze. Semiotica (1973)Google Scholar
  3. 3.
    Cassell, J., Nakano, Y., Bickmore, T., et al.: Non-Verbal Cues for Discourse Structure. In: Proceedings of the 41st Annual Meeting of the Association of Computational Linguistics, pp. 106–115 (2001)Google Scholar
  4. 4.
    Cassell, J., Sullivan, J., Prevost, S., et al. (eds.): Embodied conversational agents. MIT Press, Cambridge (2000)Google Scholar
  5. 5.
    Cassell, J., Steedman, M., Badler, N., et al.: Modeling the Interaction between Speech and Gesture (1994)Google Scholar
  6. 6.
    Cassell, J., Bickmore, T., Campbell, L., et al.: More than just a Pretty Face: Conversational Protocols and the Affordances of Embodiment. In: Knowledge-Based Systems, vol. 14, pp. 55–64. Elsevier, Amsterdam (2001)Google Scholar
  7. 7.
    Cassell, J., Vilhjalmsson, H., Bickmore, T.: BEAT: The Behavior Expression Animation Toolkit. In: SIGGRAPH, pp. 477–486. ACM Press, New York (2001)Google Scholar
  8. 8.
    Chovil, N.: Discourse-Oriented Facial Displays in Conversation. Research on Language and Social Interaction 25, 163–194 (1991)CrossRefGoogle Scholar
  9. 9.
    Fehr, B.J., Exline, R.V.: Social visual interaction: A conceptual and literature review. In: Siegman, A.W., Feldstein, S. (eds.) Nonverbal Behavior and Communication, pp. 225–326. Lawrence Erlbaum Associates, Inc., Hillsdale (1987)Google Scholar
  10. 10.
    Heylen, D., Kopp, S., Marsella, S., et al.: The Next Step Towards a Functional Markup Language. In: Proceedings of Intelligent Virtual Agents 2008. Springer, Heidelberg (2008)Google Scholar
  11. 11.
    Kendon, A.: Conducting interaction: Patterns of behavior in focused encounters. Cambridge University Press, New York (1990)Google Scholar
  12. 12.
    Kopp, S., Krenn, B., Marsella, S.C., Marshall, A.N., Pelachaud, C., Pirker, H., Thórisson, K.R., Vilhjálmsson, H.H.: Towards a Common Framework for Multimodal Generation: The Behavior Markup Language. In: Gratch, J., Young, M., Aylett, R.S., Ballin, D., Olivier, P. (eds.) IVA 2006. LNCS, vol. 4133, pp. 205–217. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  13. 13.
    Thorisson, K.: Gandalf: An Embodied Humanoid Capable of Real-Time Multimodal Dialogue with People. In: 1st International Conference on Autonomous Agents, pp. 536–537. ACM, New York (1997)Google Scholar
  14. 14.
    Vilhjálmsson, H.H., Cantelmo, N., Cassell, J., Chafai, N.E., Kipp, M., Kopp, S., Mancini, M., Marsella, S.C., Marshall, A.N., Pelachaud, C., Ruttkay, Z., Thórisson, K.R., van Welbergen, H., van der Werf, R.J.: The Behavior Markup Language: Recent Developments and Challenges. In: Pelachaud, C., Martin, J.-C., André, E., Chollet, G., Karpouzis, K., Pelé, D. (eds.) IVA 2007. LNCS, vol. 4722, pp. 99–111. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  15. 15.
    Vilhjalmsson, H., Stacy, M.: Social Performance Framework. In: Workshop on Modular Construction of Human-Like Intelligence at the 20th National AAAI Conference on Artificial Intelligence, AAAI (2005)Google Scholar
  16. 16.
    Vilhjalmsson, H.: Augmenting Online Conversation through Automatice Discourse Tagging. In: HICSS 6th Annual Minitrack on Persistent Conversation. IEEE, Los Alamitos (2005)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Hannes Högni Vilhjálmsson
    • 1
  1. 1.Center for Analysis and Design of Intelligent Agents (CADIA) School of Computer ScienceReykjavík UniversityReykjavíkIceland

Personalised recommendations