Abstract
The paper presents a formal framework to model the fusion of gesture meaning with the meaning of the co-occurring verbal fragment. The framework is based on the formalization of two simple concepts, intersectivity and iconicity, which form the core of most descriptive accounts of the interaction of the two modalities. The formalization is presented as an extension of a well-known framework for the analysis of meaning in natural language. We claim that a proper formalization of these two concepts is sufficient to provide a principled explanation of gestures accompanying different types of linguistic expressions. The formalization also aims at providing a general mechanism (iconicity) by which the meaning of gestures is extracted from their formal appearance.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Aiello, M., Pratt-Hartmann, I., van Benthem, J. (eds.): Handbook of Spatial Logics. Springer (2007)
Biederman, I.: Recognition-by-components: A theory of human image understanding. Psychological Review 94, 115–147 (1987)
Bloch, I., Heijmans, H., Ronse, C.: Mathematical morphology. In: Aiello, et al. [1], ch. 14, pp. 857–944
Carpenter, B.: Type-logical Semantics. MIT Press (1998)
Cuxac, C.: French Sign Language: Proposition of a Structural Explanation by Iconicity. In: Braffort, A., Gibet, S., Teil, D., Gherbi, R., Richardson, J. (eds.) GW 1999. LNCS (LNAI), vol. 1739, pp. 165–184. Springer, Heidelberg (2000)
van Fraassen, B.: Scientific Representation: Paradoxes and Perspectives. Oxford University Press (2008)
Giorgolo, G.: Space and Time in Our Hands, vol. 262. LOT Netherlands Graduate School of Linguistics (2010), Doctoral Dissertation
Kendon, A.: Gesture: Visible Action as Utterance. Cambridge University Press, Cambridge (2004)
Lascarides, A., Stone, M.: A formal semantic analysis of gesture. Journal of Semantics 4(26), 147–180 (2009)
Lücking, A., Bergmann, K., Hahn, F., Kopp, S., Rieser, H.: The Bielefeld Speech and Gesture Alignment Corpus (SaGA). In: Kipp, M., Martin, J.C., Paggio, P., Heylen, D. (eds.) LREC 2010 Workshop: Multimodal Corpora Advances in Capturing, Coding and Analyzing Multimodality (2010)
McNeill, D.: Hand and Mind. University of Chicago Press (1992)
Pratt-Hartmann, I.: First-order mereotopology. In: Aiello, et al. [1], ch. 2, pp. 13–98
Rieser, H.: On Factoring Out a Gesture Typology from the Bielefeld Speech-and-Gesture-Alignment Corpus (SAGA). In: Kopp, S., Wachsmuth, I. (eds.) GW 2009. LNCS, vol. 5934, pp. 47–60. Springer, Heidelberg (2010)
Streeck, J.: Depicting by gesture. Gesture 3(8), 285–301 (2008)
Varzi, A.: Spatial reasoning and ontology: Parts, wholes and locations. In: Aiello, et al. [1], ch. 15, pp. 945–1038
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Giorgolo, G. (2012). Integration of Gesture and Verbal Language: A Formal Semantics Approach. In: Efthimiou, E., Kouroupetroglou, G., Fotinea, SE. (eds) Gesture and Sign Language in Human-Computer Interaction and Embodied Communication. GW 2011. Lecture Notes in Computer Science(), vol 7206. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-34182-3_20
Download citation
DOI: https://doi.org/10.1007/978-3-642-34182-3_20
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-34181-6
Online ISBN: 978-3-642-34182-3
eBook Packages: Computer ScienceComputer Science (R0)