Skip to main content
SpringerLink
Log in

Speaking without Thinking: Embodiment, Speech Technology and Social Signal Processing

  • Chapter
Toward Autonomous, Adaptive, and Context-Aware Multimodal Interfaces. Theoretical and Practical Issues

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 6456))

  • 1219 Accesses

Abstract

Neuroimaging results and related results from neuroscience have shown that the “multimodal” areas of the brain responsible for sensory integration are fundamental to semantic comprehension and language production. Developmental studies have shown that language comprehension and production involves infants learning by imitation from their caregivers to coordinate multimodal schemas that cross from one perceptual modality to another. Analyzing the both the physiological and socio-cultural constraints of embodiment on language, perception and cognition, I develop a theory of image schemas. As with the Gestalt perceptual structures, image schemas are dynamic perceptual wholes which prompt for ‘normal’ pattern completions based on our recurrent experiences in interacting with the world. I conclude by describing how both the theory of image schemas and embodied cognitive science generally can bring a fresh perspective to machine-learning problems such as the visual recognition of speech, emotion and gesture.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Turing, A.M.: Computing machinery and intelligence. Mind 59, 433–460 (1950)

    Article  MathSciNet  Google Scholar 

  2. Bohn, O.: Linguistic relativity in speech perception: An overview of the influence of language experience on the perception of speech sounds from infancy to adulthood. In: Niemeier, S., Dirven, R. (eds.) Evidence for Linguistic Relativity, pp. 1–12. John Benjamins, Amsterdam (2000)

    Chapter  Google Scholar 

  3. Kristiansen, G.: How to do things with allophones. Linguistic stereotypes as cognitive reference points in social cognition. In: Dirven, R., Frank, R., Pütz, M. (eds.) Cognitive Models in Language and Thought. Ideology, Metaphors and Meaning, pp. 69–120. Mouton de Gruyter, Berlin (2003)

    Google Scholar 

  4. Meltzoff, A., Moore, M.K.: Imitation of facial and manual gestures by human neonates. Science 198, 74–78 (1977)

    Article  Google Scholar 

  5. Meltzoff, A.: Molyneux’s babies: Cross-modal perception, imitation and the mind of the preverbal infant. In: Spatial Representation: Problems in Philosophy and Psychology, pp. 219–235. Blackwell, Cambridge (1993)

    Google Scholar 

  6. Johansson, G.: Visual perception of biological motion and a model for its analysis. Perception and Psychophysics 14, 201–211 (1973)

    Article  Google Scholar 

  7. Wertheimer, M.: Laws of organization in perceptual forms. In: Ellis, W (ed. & trans.), A source book of Gestalt psychology, pp. 71-88. Routledge & Kegan Paul, London (Original work published in 1923, as Untersuchungen zur Lehre von der Gestalt II, in Psychologische Forschung, 4:301-350) (1923/1938)

    Google Scholar 

  8. Arterberry, M.E., Borsnstein, M.H.: Three-month-old infants’ categorization of animals and vehicles based on static and dynamic attributes. J. Experimental Child Psychology 80, 333–346 (2001)

    Article  Google Scholar 

  9. Bertenthal, B.I.: Infants’ perception of biomechanical motions: Intrinsic image and knowledge-based constraints. In: Granrud, C. (ed.) Visual Perception and Cognition in Infancy, pp. 175–214. Routledge, New York (1993)

    Google Scholar 

  10. Meltzoff, A., Borton, R.W.: Intermodal matching by human neonates. Nature 282, 403–404 (1979)

    Article  Google Scholar 

  11. Stern, D.N.: The interpersonal world of the infant. Basic Books, New York (1985)

    Google Scholar 

  12. Rose, S.A., Ruff, H.A.: Cross modal abilities in human infants. In: Osofsky, J.D. (ed.) Handbook of infant development, pp. 318–362. Wiley, New York (1987)

    Google Scholar 

  13. Rose, S.A., Blank, M.S., Bridger, W.H.: Intermodal and Intramodal retention of visual and tactual information in young children. Developmental Psychology 6, 482–486 (1972)

    Article  Google Scholar 

  14. Lewkowicz, D.J., Turkewitz, G.: Intersensory interaction in newborns: modification of visual preferences following exposure to sound. Child Development 52, 827–832 (1981)

    Article  Google Scholar 

  15. Woodward, A.L.: Infants selectively encode the goal object of an actor’s reach. Cognition 69, 1–34 (1998)

    Article  Google Scholar 

  16. Woodward, A.L.: Infants’ ability to distinguish between purposeful and non-purposeful behaviors. Infant Behavior and Development 22, 145–160 (1999)

    Article  Google Scholar 

  17. Woodward, A.L., Guajardo, J.J.: Infants’ understanding of the point gesture as an object-directed action. Cognitive Development 83, 1–24 (2002)

    Google Scholar 

  18. Johnson, M.L.: The Body in the Mind: The Bodily Basis of Meaning, Imagination and Reason. University of Chicago Press, Chicago (1987)

    Google Scholar 

  19. Lakoff, G.: Women, Fire and Dangerous Things. University of Chicago Press, Chicago (1987)

    Book  Google Scholar 

  20. Lakoff, G., Johnson, M.L.: Philosophy in the Flesh: The Embodied Mind and Its Challenge to Western Thought. Basic Books, New York (1999)

    Google Scholar 

  21. Johnson, M.L., Rohrer, T.: We are live creatures: Embodiment, American pragmatism, and the cognitive organism. In: Zlatev, J., Ziemke, T., Frank, R., Dirven, R. (eds.) Body, Language, and Mind, vol. 1, pp. 17–54. Mouton de Gruyter, Berlin (2007)

    Google Scholar 

  22. Rizzolatti, G., Craighero, L.: The mirror neuron system. Annual Review of Neuroscience 27, 169–192 (2004)

    Article  Google Scholar 

  23. Fogassi, L., Gallese, V., Buccino, G., Craighero, L., Fadiga, L., Rizzolatti, G.: Cortical mechanism for the visual guidance of hand grasping movements in the monkey: A reversible inactivation study. Brain 124, 571–586 (2001)

    Article  Google Scholar 

  24. Buccino, G., Binkofski, F., Fink, G.R., Fadiga, L., Fogassi, L., Gallese, V., Seitz, J., Zilles, K., Rizzolatti, G., Freund, H.: Action observation activates premotor and parietal areas in a somatotopic manner: an fMRI study. European Journal of Neuroscience 13, 400–404 (2001)

    Google Scholar 

  25. Umiltá, M.A., Kohler, E., Gallese, V., Fogassi, L., Fadiga, L., Keysers, C., Rizzolatti, G.I.: know what you are doing: A neurophysiological study. Neuron 31, 155–165 (2001)

    Article  Google Scholar 

  26. Ferrari, P.F., Gallese, V., Rizzolatti, G., Fogassi, L.: Mirror neurons responding to the observation of ingestive and communicative mouth actions in the monkey ventral premotor cortex. European Journal of Neuroscience 17, 1703–1714 (2003)

    Article  Google Scholar 

  27. Kohler, E., Keysers, C., Umiltá, M.A., Fogassi, L., Gallese, V., Rizzolatti, G.: Hearing sounds, understanding actions: action representation in mirror neurons. Science 297, 846–848 (2002)

    Article  Google Scholar 

  28. Suzuki, K., Yamadori, A., Fujii, T.: Category specific comprehension deficit restricted to body parts. Neurocase 3, 193–200 (1997)

    Article  Google Scholar 

  29. Penfield, W.G., Rasmussen, T.B.: The cerebral cortex of man. Macmillan, New York (1950)

    Google Scholar 

  30. Coslett, H.B., Saffran, E.M., Schwoebel, J.: Knowledge of the human body: a distinct semantic domain. Neurology 59, 357–363 (2002)

    Article  Google Scholar 

  31. Schwoebel, J., Coslett, H.B.: Evidence for multiple, distinct representations of the human body. Journal of Cognitive Neuroscience 4, 543–553 (2005)

    Article  Google Scholar 

  32. Hauk, O., Johnsrude, I., Pulvermüller, F.: Somatotopic representation of action words in human motor and premotor cortex. Neuron 41, 301–307 (2004)

    Article  Google Scholar 

  33. Pulvermüller, F., Hauk, O., Nikulin, V., Ilmoniemi, R.J.: Functional interaction of language and action processing: A TMS study. MirrorBot: Biometric multimodal learning in a mirror neuron-based robot, Report #8 (2002)

    Google Scholar 

  34. Rohrer, T.: Image Schemata in the Brain. In: Hampe, B. (ed.) From Perception to Meaning: Image Schemas in Cognitive Linguistics, pp. 165–196. Mouton de Gruyter, Berlin (2005)

    Chapter  Google Scholar 

  35. Zwaan, R.A., Madden, C.J., Yaxley, R.A., Aveyard, M.A.: Moving words: Dynamic representations in language comprehension. Cognitive Science 28, 611–619 (2004)

    Google Scholar 

  36. Glenberg, A.M., Kaschak, M.P.: Grounding language in action. Psychonomic Bulletin & Review 9, 558–565 (2002)

    Article  Google Scholar 

  37. Gibbs, R.W.: The psychological status of image schemas. In: Hampe, B. (ed.) From Perception to Meaning: Image Schemas in Cognitive Linguistics, pp. 113–135. Mouton de Gruyter, Berlin (2005)

    Chapter  Google Scholar 

  38. Rohrer, T.: Understanding through the body: fMRI and ERP studies of metaphoric and literal language. Paper presented at the 7th International Cognitive Linguistics Association conference (2001)

    Google Scholar 

  39. Moore, C.I., Stern, C.I., Corkin, S., Fischl, B., Gray, A.C., Rosen, B.R., Dale, A.M.: Segregation of somatosensory activation in the human rolandic cortex using fMRI. Journal of Neurophysiology 84, 558–569 (2000)

    Google Scholar 

  40. Matlock, T., Ramscar, M., Boroditsky, L.: The experiential link between spatial and temporal language. Cognitive Science 29, 655–664 (2005)

    Article  Google Scholar 

  41. Fischl, B., Sereno, M.I., Tootell, R.B.H., Dale, A.M.: High-resolution inter-subject averaging and a coordinate system for the cortical surface. Human Brain Mapping 8, 272–284 (1999)

    Article  Google Scholar 

  42. McGurk, H., MacDonald, J.: Hearing lips and seeing voices. Nature, 746–748 (1976)

    Google Scholar 

  43. Rohrer, T.: The body in space: Embodiment, experientialism and linguistic conceptualization. In: Zlatev, J., Ziemke, T., Frank, R., Dirven, R. (eds.) Body, Language, and Mind, vol. 2, pp. 339–378. Mouton de Gruyter, Berlin (2007)

    Google Scholar 

  44. Rohrer, T.: Pragmatism, ideology and embodiment: William James and the philosophical foundations of cognitive linguistics. In: Dirven, R., Hawkins, B., Sandikcioglu, E. (eds.) Language and Ideology: Cognitive Theoretic Approaches, vol. 1, pp. 49–81. John Benjamins, Amsterdam (2001)

    Chapter  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Colorado Advanced Research Institute, Box 3736, Boulder, Colorado, 80307-3736

    Tim Rohrer

Authors
  1. Tim Rohrer
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Institute for Advanced Scientific Studies, Second University of Naples, and IIASS, Via Pellegrino 19, 84019, Vietri sul Mare (SA), Italy

    Anna Esposito

  2. Istituto Nazionale di Geofisica e Vulcanologia, Osservatorio Vesuviano, Via Diocleziano 328, 80124, Napoli, Italy

    Antonietta M. Esposito

  3. Dipartemento di Ingegneria dell’ Informazione, Seconda Università di Napoli, Via Roma 29, 81031, Aversa (CE), Italy

    Raffaele Martone

  4. Department of Humanities and Social Sciences, Anatolia College/ACT, Kennedy Street, 55510, Pylaia, Greece

    Vincent C. Müller

  5. Departmnet of Physics "E.R. Caoamoeööp", University of Salerno and IIASS, International Institute for Advanced Scientific Studies, 84081, Baronissi (SA), Italy

    Gaetano Scarpetta

Rights and permissions

Reprints and permissions

Copyright information

© 2011 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Rohrer, T. (2011). Speaking without Thinking: Embodiment, Speech Technology and Social Signal Processing. In: Esposito, A., Esposito, A.M., Martone, R., Müller, V.C., Scarpetta, G. (eds) Toward Autonomous, Adaptive, and Context-Aware Multimodal Interfaces. Theoretical and Practical Issues. Lecture Notes in Computer Science, vol 6456. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-18184-9_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-18184-9_2

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-18183-2

  • Online ISBN: 978-3-642-18184-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation