Skip to main content
SpringerLink
Log in

Gesture, Prosody and Lexicon in Task-Oriented Dialogues: Multimedia Corpus Recording and Labelling

  • Conference paper
Verbal and Nonverbal Communication Behaviours

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 4775))

Abstract

The aim of the DiaGest Project is to study interdependencies between gesture, lexicon, and prosody in Polish dialogues. The material under study comprises three tasks realised by twenty pairs of subjects. Two tasks involve instructional, task-oriented dialogues, while the third is based on a question answering procedure. A system for corpus labelling is currently being designed on the basis of current standards. The corpus will be annotated for gestures, lexical content of utterances, intonation and rhythm. In order to relate various phenomena to the contextualized meaning of dialogue utterances, the material will also be tagged in terms of dialogue acts. Synchronised tags will be placed in respective annotation tiers in ELAN. A number of detailed studies related to the problems of gesture-prosody, gesture-lexicon and prosody-lexicon interactions will be carried out on the basis of the tagged material.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Alexandersson, J., Buschbeck-Wolf, B., Fujinami, T., Kipp, M., Koch, S., Maier, E., Reithinger, N., Schmitz, B.: Dialogue Acts in VERBMOBIL-2, 2nd edn. (Deliverable) (1998)

    Google Scholar 

  2. Allwood, J., Cerrato, L., Dybkjaer, L., Jokinen, K., Navaretta, C., Paggio, P.: The MUMIN Multimodal Coding Scheme. NorFA Yearbook. (2005)

    Google Scholar 

  3. Antas, J.: Gest, mowa a mysl. In: Grzegorczykowa R., Pajdzinska A. (eds.) Jezykowa kategoryzacja swiata. Lublin (1996)

    Google Scholar 

  4. Antas, J.: Morfologia gestu. Rozwazania metodologiczne. In: Slawski F., Mieczkowska H. (eds.) Studia z jezykoznawstwa slowianskiego. Krakow (1995)

    Google Scholar 

  5. Antas, J.: Co mowia rece. Wprowadzenie do komunikacji niewerbalnej. In: Przybylska R., Przyczyna W. (eds.) Retoryka dzis. Teoria i praktyka. Krakow (2001)

    Google Scholar 

  6. Boersma, P., Wenink, D.: Praat. Doing Phonetics by Computer (a computer program; version 4.4 and later) (2006)

    Google Scholar 

  7. Bolinger, D.: Intonation and Gesture. American Speech 58(2), 156–174 (1983)

    Article  Google Scholar 

  8. Bunt, H.: A Framework for Dialogue Act Specification. In: Paper presented at the 4th Joint ISO-SIGSEM Workshop on the Representation of Multimodal Semantic Information, Tilburg (2005)

    Google Scholar 

  9. Bunt, H.C., Girard, Y.M.: Designing an Open, Multidimensional Dialogue Act Taxonomy. In: Gardent, C., Gaiffe, B. (eds.) DIALOR 2005. Proceedings of the Ninth International Workshop on the Semantics and Pragmatics of Dialogue, pp. 37–44 (2006)

    Google Scholar 

  10. Carletta, J., Isard, A., Isard, S., Kowtko, J., Doherty-Sneddon, J., Anderson, A.: HCRC: Dialogue Structure Coding Manual, Human Communications Research Centre. University of Edinburgh, Edinburgh, HCRC TR – 82 (1996)

    Google Scholar 

  11. Cole, R.A., Carmell, T., Connors, P., Macon, M., Wouters, J., de Villiers, J., Tarachow, A., Massaro, D., Cohen, M., Beskow, J., Yang, J., Meier, U., Waibel, A., Stone, P., Fortier, G., Davis, A., Soland, C.: Intelligent Animated Agents for Interactive Language Training. In: STiLL: ESCA Workshop on Speech Technology in Language Learning. Stockholm, Sweden (1998)

    Google Scholar 

  12. Cole, R.A., Van Vuuren, S., Pellom, B., Hacioglu, K., Ma, J., Movellan, J., Schwartz, S., Wade-Stein, D., Ward, W., Yan, J.: Perceptive Animated Interfaces: First Steps Toward a New Paradigm for Human–Computer Interaction. Proceedings of the IEEE: Special Issue on Human-Computer Multimodal Interface 91(9), 1391–1405 (2003)

    Google Scholar 

  13. Core, M., Allen, J.: Coding Dialogues with the DAMSL Annotation Scheme. In: AAAI Fall Symposium on Communicative Action in Humans and Machines, Cambridge, MA, pp. 28–35 (1997)

    Google Scholar 

  14. Demenko, G., Wypych, M., Baranowska, E.: Implementation of Grapheme-to-phoneme Rules and Extended SAMPA Alphabet in Polish Text-to-speech Synthesis. Speech and Language Technology 7, 17. Wydawnictwo PTFon, Poznan (2003)

    Google Scholar 

  15. Dilley, L., Breen, M., Bolivar, M., Kraemer, J., Gibson, E.: A Comparison of Inter-Transcriber Reliability for Two Systems of Prosodic Annotation: RaP (Rhythm and Pitch) and ToBI (Tones and Break Indices). In: Proceedings of the International Conference on Spoken Language Processing, INTERSPEECH 2006, Pittsburgh, PA (2006)

    Google Scholar 

  16. Dilley, L., Brown, M.: The RaP Labeling System, v. 1.0, ms (2005), http://faculty.psy.ohio-state.edu/pitt/dilley/rapsystem.htm

  17. Dziubalska-Kolaczyk, K., Krynicki, G., Sobkowiak, W., Bogacka, A., et al.: The Use of Metalinguistic Knowledge in a Polish Literacy Tutor. In: Duszak, A., Okulska, U. (eds.) GlobE 2004. Peter Lang (2004)

    Google Scholar 

  18. Francuzik, K., Karpinski, M., Klesta, J., Szalkowska, E.: Nuclear Melody in Polish Semi-Spontaneous and Read Speech: Evidence from Polish Intonational Database PoInt. Studia Phonetica Posnanensia 7, 97–128 (2005)

    Google Scholar 

  19. Garcia, J., Gut, U., Galves, A.: Vocale: A Semi-automatic Annotation Tool for Prosodic Research. In: Proceedings of Speech Prosody, Aix-en-Provence 2002, pp. 327–330 (2002)

    Google Scholar 

  20. Gibbon, D., Mertins, I., Moore, R.K. (eds.): Handbook of Multimodal and Spoken Dialogue Systems: Resources, Terminology and Product Evaluation. Kluwer Academic Publishers, Dordrecht (2000)

    Google Scholar 

  21. Gibbon, D., Moore, R.K., Winsky, R(eds.): The Eagles Handbook of Standards and Resources for Spoken Language Systems. Mouton de Gruyter (1997)

    Google Scholar 

  22. Gut, U., Looks, K., Thies, A., Gibbon, D.: CoGesT: Conversational Gesture Transcription System. Version 1.0. Technical report. Bielefeld University (2003)

    Google Scholar 

  23. Hellwig, B., Uytvanck, D.: EUDICO Linguistic Annotator: ELAN, Version 3.0 Manual software manual (2004)

    Google Scholar 

  24. Hirst, D.J., Di Cristo, A., Espesser, R.: Levels of Representation and Levels of Analysis for Intonation. In: Horne, M. (ed.) Prosody: Theory and Experiment, Kluwer, Dordrecht (2000)

    Google Scholar 

  25. Hirst, D., Espesser, R.: Automatic Modelling of Fundamental Frequency Using a Quadratic Spline Function. Travaux de l’Institut de Phonétique d’Aix-en-Provence 15, 71–85 (1993)

    Google Scholar 

  26. Jannedy, S., Mendoza-Denton, N.: Structuring Information through Gesture and Intonation. In: Ishihara, S., Schmitz, M., Schwarz, A. (eds.) Interdisciplinary Studies on Information Structure 03, pp. 199–244 (2005)

    Google Scholar 

  27. Jassem, W.: Classification and Organization of Data in Intonation Research. In: Braun, A., Masthoff, H.R. (eds.) Phonetics and its Applications. Festschrift for Jens-Peter Köster. Franz Steiner Verlag, Wiesbaden, pp. 289–297 (2002)

    Google Scholar 

  28. Karpinski, M.: Struktura i intonacja polskiego dialogu zadaniowego. Wydawnictwo Naukowe UAM, Poznan (2006)

    Google Scholar 

  29. Kendon, A.: Gesticulation and Speech: two Aspects of the Process. In: Key, M.R. (ed.) The Relation Between Verbal and Nonverbal Communication, Mouton (1980)

    Google Scholar 

  30. Kendon, A.: Gesture and Speech: How They Interact. In: Wiemann, J.M., Harrison, R.P. (eds.) Nonverbal Interaction, pp. 13–43. Sage Publications, Beverly Hills (1983)

    Google Scholar 

  31. Kipp, M.: Anvil: A Generic Annotation Tool for Multimodal Dialogue. In: Proceedings of the 7th European Conference on Speech Communication and Technology, EUROSPEECH 2001, Aalborg pp. 1367–1370 (2001)

    Google Scholar 

  32. Kipp, M., Neff, M., Albrecht, I.: An Annotation Scheme for Conversational Gestures: How to Economically Capture Timing and Form. In: Martin, J.-C., Kühnlein, P., Paggio, P., Stiefelhagen, R., Pianesi, F. (eds.) LREC 2006 Workshop on Multimodal Corpora: From Multimodal Behaviour Theories to Usable Models (2006)

    Google Scholar 

  33. Kita, S., van Gijn, I., van der Hulst, H.: Movement Phases in Signs and Co-speech Gestures and Their Transcription by Human Coders. In: Wachsmuth, I., Fröhlich, M. (eds.) Gesture and Sign Language in Human-Computer Interaction, pp. 23–35. Springer, Heidelberg (1998)

    Chapter  Google Scholar 

  34. Klein, M.: Standardisation Efforts on the Level of Dialogue Acts in the MATE Project. In: Proceedings of the ACL Workshop: Towards Standards and Tools for Discourse Tagging. University of Maryland, pp. 35–41 (1999)

    Google Scholar 

  35. Loehr, D.: Gesture and Intonation. Doctoral Dissertation, Georgetown University, Washington, DC (2004)

    Google Scholar 

  36. Louw, J.A., Barnard, E.: Automatic Intonation Modelling with INTSINT. In: Proceedings of the Fifteenth Annual Symposium of the Pattern Recognition Association of South Africa, UCT Press, pp. 107–111 (2004)

    Google Scholar 

  37. Malandro, L.A., Barker, L.L., Barker, D.A.: Nonverbal Communication. Addison-Wesley, Reading, MA (1989)

    Google Scholar 

  38. Martell, C.: FORM: An Extensible, Kinematically-Based Gesture Annotation Scheme. In: Proceedings of ICSLP 2002, Denver, Colorado, pp. 353–356 (2002)

    Google Scholar 

  39. Mengel, A., Dybkjaer, L., Garrido, J.M., Heid, U., Klein, M., Pirrelli, V., Poesio, M., Quazza, S., Schiffrin, A., Soria, C.: MATE: Deliverable D2.1 MATE Dialogue Annotation Guidelines (2000)

    Google Scholar 

  40. Mertens, P.: The Prosogram: Semi-Automatic Transcription of Prosody Based on a Tonal Perception Model. In: Bel, B., Marlien, I. (eds.) Proceedings of Speech Prosody 2004, Nara, Japan (2004)

    Google Scholar 

  41. McNeill, D.: Hand and Mind: What Gestures Reveal about Thought. University of Chicago Press, Chicago (1992)

    Google Scholar 

  42. Prillwitz, S., Leven, R., Zienert, H., Hanke, T., Henning, J.: HamNoSys. Version 2.0. Hamburg Notation System for Sign Languages. An Introductory Guide. Signum, Hamburg (1989)

    Google Scholar 

  43. Przepiorkowski, A., Wolinski, M.: A Flexemic Tagset for Polish. In: The Proceedings of the Workshop on Morphological Processing of Slavic Languages, EACL 2003 (2003)

    Google Scholar 

  44. Silverman, K., Beckman, M., Pierrehumbert, J., Ostendorf, M., Wightman, C., Price, P., Hirschberg, J.: ToBI: A Standard Scheme for Labeling Prosody. In: Proceedings of ICSLP, pp. 867–869 (1992)

    Google Scholar 

  45. Steffen-Batogowa, M.: Struktura przebiegu melodii jezyka polskiego ogolnego. Poznan (1996)

    Google Scholar 

  46. Steininger, S., Schiel, F., Louka, K.: Gestures During Overlapping Speech in Multimodal Human-Machine Dialogues. In: International Workshop on Information Presentation and Natural Multimodal Dialogue 2001, Verona, Italy (2001)

    Google Scholar 

  47. Swerts, M., Krahmer, E.: The Effects of Visual Beats on Prosodic Prominence. In: Proceedings of Speech Prosody 2006, Dresden (2006)

    Google Scholar 

  48. Valbonesi, L., Ansari, R., McNeill, D., Quek, F., Duncan, S., McCullough, K., et al.: Multimodal Signal Analysis of Prosody and Hand Motion: Temporal Correlation of Speech and Gestures. In: EUSIPCO 2002. European Signal Processing Conference (2002)

    Google Scholar 

  49. Wolinski, M.: System znacznikow morfosyntaktycznych w korpusie IPI PAN. Polonica XXII-XXIII, pp. 39–55 (2003)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Centre for Speech and Language Processing, Adam Mickiewicz University, Collegium Novum, Al. Niepodleglosci 4, 61-784 Poznan, Poland

    Ewa Jarmolowicz, Maciej Karpinski, Zofia Malisz & Michal Szczyszek

Authors
  1. Ewa Jarmolowicz
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Maciej Karpinski
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zofia Malisz
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Michal Szczyszek
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Anna Esposito Marcos Faundez-Zanuy Eric Keller Maria Marinaro

Rights and permissions

Reprints and permissions

Copyright information

© 2007 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Jarmolowicz, E., Karpinski, M., Malisz, Z., Szczyszek, M. (2007). Gesture, Prosody and Lexicon in Task-Oriented Dialogues: Multimedia Corpus Recording and Labelling. In: Esposito, A., Faundez-Zanuy, M., Keller, E., Marinaro, M. (eds) Verbal and Nonverbal Communication Behaviours. Lecture Notes in Computer Science(), vol 4775. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-76442-7_9

Download citation

  • DOI: https://doi.org/10.1007/978-3-540-76442-7_9

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-76441-0

  • Online ISBN: 978-3-540-76442-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation