Advertisement

Multimodal Behaviours in Comparable Danish and Polish Human-Human Triadic Spontaneous Interactions

  • Costanza Navarretta
  • Magdalena Lis
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8511)

Abstract

This is a pilot study of multimodal behaviours in manually annotated comparable video recordings of Danish and Polish triadic naturally occurring conversations. The data are comparable with respect to the conversational settings, the familiarity degree, age and gender of the participants. Furthermore, they have been annotated according to the same annotation scheme following common coding strategies. The analysis of the annotations indicates that although the conversations in the two languages differ in content, Danes and Poles use the same type of head movements and with the same frequency. In both datasets the most common facial expressions are laughter and smile, however, facial expressions are much more frequent in the Polish data than in the Danish data. Furthermore, the facial expressions in the Polish data are often used as feedback signals to the interlocutors while the Danes use facial expression to comment their own spoken contribution. Finally, the Danes use more frequently hand gestures than the Poles and their hand gestures have a deictic function while the hand gestures of the Poles are iconic. The differences in the behaviours in the two corpora can partly depend on the language, but is also due to the type of relationship between the participants and the content of the conversations.

Keywords

Multimodal Corpora Multilinguality Human-human Interaction 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    McClave, E.: Linguistic functions of head movements in the context of speech. Journal of Pragmatics 32, 855–878 (2000)CrossRefGoogle Scholar
  2. 2.
    Cerrato, L.: Investigating Communicative Feedback Phenomena across Languages and Modalities. PhD thesis, Stockholm, KTH, Speech and Music Communication (2007)Google Scholar
  3. 3.
    Paggio, P., Navarretta, C.: Head movements, facial expressions and feedback in Danish first encounters interactions: A culture-specific analysis. In: Stephanidis, C. (ed.) Universal Access in HCI, Part II, HCII 2011. LNCS, vol. 6766, pp. 583–590. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  4. 4.
    Maynard, S.: Interactional functions of a nonverbal sign: Head movement in Japanese dyadic casual conversation. Journal of Pragmatics 11, 589–606 (1987)CrossRefGoogle Scholar
  5. 5.
    Jokinen, K., Navarretta, C., Paggio, P.: Distinguishing the communicative functions of gestures. In: Popescu-Belis, A., Stiefelhagen, R. (eds.) MLMI 2008. LNCS, vol. 5237, pp. 38–49. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  6. 6.
    Rehm, M., Nakano, Y., André, E., Nishida, T.: Culture-Specific First Meeting Encounters between Virtual Agents. In: Prendinger, H., Lester, J.C., Ishizuka, M. (eds.) IVA 2008. LNCS (LNAI), vol. 5208, pp. 223–236. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  7. 7.
    Lu, J., Allwood, J., Ahlsén, E.: A study on cultural variations of smile based on empirical recordings of Chinese and Swedish first encounters. In: Heylen, D., Kipp, M., Paggio, P. (eds.) Proceedings of the Workshop on Multimodal Corpora at ICMI-MLMI 2011, Alicante, Spain (2011)Google Scholar
  8. 8.
    Navarretta, C., Ahlsn, E., Allwood, J., Jokinen, K., Paggio, P.: Feedback in nordic first-encounters: a comparative study. In: Proceedings of LREC 2012, Istanbul, Turkey, pp. 2494–2499 (May 2012)Google Scholar
  9. 9.
    Jacobs, N., Garnham, A.: The role of conversational hand gestures in a narrative task. Journal of Memory and Language 56(2), 291–303 (2007)CrossRefGoogle Scholar
  10. 10.
    Kita, S., Özyurek, A.: How does spoken language shape iconic gestures? In: Duncan, S., Cassel, J., Levy, E. (eds.) Gesture and the Dynamic Dimension of Language, pp. 67–74. Benjamins, Amsterdam (2007)Google Scholar
  11. 11.
    Gullberg, M.: Thinking, speaking and gesturing about motion in more than one language. In: Pavlenko, A. (ed.) Thinking and Speaking in two Languages, pp. 143–169 (2011)Google Scholar
  12. 12.
    Lis, M., Parrill, F.: Referent type and its verbal and gestural representation: A test on English multimodal corpus and WordNet 3.1. In: Extended Abstract in Proceedings of the 1st European Symposium on Multimodal Communication, MMSym 2013 (2013)Google Scholar
  13. 13.
    McNeill, D.: Hand and mind: What gestures reveal about thought. University of Chicago Press, Chicago (2000)Google Scholar
  14. 14.
    Goldin-Meadow, S., Chee So, W., Ozyurek, A., Mylander, C.: The natural order of events: How speakers of different languages represent events nonverbally. Proceedings of the National Academy of Sciences of the USA 105(27), 9163–9168 (2008)CrossRefGoogle Scholar
  15. 15.
    Navarretta, C., Paggio, P.: Verbal and non-verbal feedback in different types of interactions. In: Proceedings of LREC 2012, Istanbul Turkey, pp. 2338–2342 (2012)Google Scholar
  16. 16.
    Campbell, N.: Individual Traits of Speaking Style and Speech Rhythm in a Spoken Discourse. In: Proceedings of the COST 2007 Workshop, pp. 107–120 (2007)Google Scholar
  17. 17.
    Navarretta, C., Lis, M.: Multimodal feedback expressions in Danish and Polish spontaneous conversations. In: Allwood, J., et al. (eds.) NEALT Proceedings. Northern European Association for Language and Technology, Proceedings of the Fourth Nordic Symposium of Multimodal Communication, Gothenburg, Sweden, November 2012. Linköping Electronic Conference Proceedings, pp. 55–62 (2013)Google Scholar
  18. 18.
    VYngve, V.: On getting a word in edgewise. Papers from the Sixth Regional Meeting of the Chicago Linguistic Society, pp. 567–578 (1970)Google Scholar
  19. 19.
    Rehm, M., et al.: Creating Standardized Video Recordings of Multimodal Interactions across Cultures. In: Kipp, M., Martin, J.-C., Paggio, P., Heylen, D., et al. (eds.) Multimodal Corpora. LNCS (LNAI), vol. 5509, pp. 138–159. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  20. 20.
    MacWhinney, B., Wagner, J.: Transcribing, searching and data sharing: The CLAN software and the TalkBank data repository. Gespraechsforschung 11, 154–173 (2010)Google Scholar
  21. 21.
    Navarretta, C.: Annotating Behaviours in Informal Interactions. In: Esposito, A. (ed.) Communication and Enactment 2010. LNCS, vol. 6800, pp. 309–315. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  22. 22.
    Navarretta, C.: Anaphora and gestures in multimodal communication. In: Hendrickx, B., Devi, L., Mitkov (eds.) Proceedings of the 8th Discourse Anaphora and Anaphor Resolution Colloquium (DAARC 2011), Edicoes Colibri, Faro, Portugal, pp. 171–181 (2011)Google Scholar
  23. 23.
    Allwood, J., Cerrato, L., Jokinen, K., Navarretta, C., Paggio, P.: The mumin coding scheme for the annotation of feedback, turn management and sequencing. Multimodal Corpora for Modelling Human Multimodal Behaviour. Special Issue of the International Journal of Language Resources and Evaluation 41(3-4), 273–287 (2007)Google Scholar
  24. 24.
    Boersma, P., Weenink, D.: Praat: Doing phonetics by computer (2013), retrieved from http://www.praat.org/
  25. 25.
    Kipp, M.: Gesture generation by imitation - from human behavior to computer character animation. Ph.D. thesis, Saarland University, Saarbruecken, Germany, Boca Raton, Florida, dissertation.com (2004)Google Scholar
  26. 26.
    Peirce, C.S.: Collected Papers of Charles Sanders Peirce, 8 vols. Harvard University Press, Cambridge (1931, 1958)Google Scholar
  27. 27.
    Cohen, J.: A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20(1), 37–46 (1960)CrossRefGoogle Scholar
  28. 28.
    Campbell, N.: An audio-visual approach to measuring discourse synchrony in multimodal conversation data. In: Proceedings of Interspeech 2009, pp. 12–14 (2009)Google Scholar
  29. 29.
    Jokinen, K., Pärkson, S.: Synchrony and copying in conversational interactions. In: Paggio, P., Ahlsén, E., Allwood, J., Jokinen, K., Navarretta, C. (eds.) Proceedings of the 3rd Nordic Symposium on Multimodal Communication. NEALT Proceedings Series, vol. 15, pp. 18–24 (2011)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Costanza Navarretta
    • 1
  • Magdalena Lis
    • 1
  1. 1.University of CopenhagenCopenhagen SDenmark

Personalised recommendations