How the Brain Processes Language in Different Modalities

  • Bencie Woll
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5398)


Establishing which neural systems support processing of sign languages informs a number of important neuroscience and linguistic questions. In this chapter, the linguistic structure of sign languages is introduced with a discussion of common myths about sign languages. This is followed by a more detailed discussion of the linguistics of British Sign Language, with special reference to features which resemble or contrast with spoken languages. The final section describes language and the brain by describing a number of neuroimaging studies with signers and research on signers who have aphasia or other language deficits following strokes. The neuroimaging and aphasia data are used to explore the ‘core language system’ - the regions of the brains used for language regardless of modality.


sign language neuroimaging language and brain modality 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Meier, R.P.: Why Different, Why the Same? Explaining Effects and Non-effects of Modality upon Linguistic Structure in Sign and Speech. In: Meier, R.P., Cormier, K., Quinto-Pozos, D. (eds.) Modality and Structure in Signed and Spoken Languages, pp. 1–25. Cambridge University Press, Cambridge (2002)CrossRefGoogle Scholar
  2. 2.
    Pinker, S., Bloom, P.: Natural Language and Natural Selection. Behav. Br Sci. 13/4, 707–784 (1990)CrossRefGoogle Scholar
  3. 3.
    Stokoe, W.C.: Sign language structure. An Outline of the visual communication system of the American deaf. Studies in Linguistics. Occasional Papers 8. University of Buffalo, Buffalo, NY (1960); revised edition. Linstok Press, Silver Spring (1978) Google Scholar
  4. 4.
    Sutton-Spence, R.L., Woll, B.: The linguistics of BSL: An introduction. Cambridge University Press, Cambridge (1999)Google Scholar
  5. 5.
    Pizzuto, E., Ardito, B., Caselli, M.C., Volterra, V.: Cognition and Language in Italian Deaf Preschoolers of Deaf and Hearing Families. In: Clark, M.D., Marschark, M., Karchmer, M.A. (eds.) Context, Cognition, and Deafness: An Introduction, pp. 49–70. Gallaudet University Press, Washington (2001)Google Scholar
  6. 6.
    Morgan, G., Herman, R., Woll, B.: The Development of Complex Verb Constructions in BSL. J. Ch. Lang 29, 655–675 (2002)Google Scholar
  7. 7.
    Tolar, T.D., Lederberg, A.R., Gokhale, S., Tomasello, M.: The Development of the Ability to Recognize the Meaning of Iconic Signs. J. Deaf. Stud. Deaf Ed. 13(2), 225–240 (2008)CrossRefGoogle Scholar
  8. 8.
    Campbell, R., Martin, P., White, T.: Forced Choice Recognition of Sign in Novice Learners of British Sign Language. App. Ling 13(2), 185–201 (1992)CrossRefGoogle Scholar
  9. 9.
    Liddell, S.K.: American Sign Language Syntax. The Hague, Mouton (1980)Google Scholar
  10. 10.
    Zeshan, U.: Interrogative constructions in signed languages: crosslinguistic perspectives. Language 80(1), 7–39 (2004)CrossRefGoogle Scholar
  11. 11.
    Jackendoff, R.: The Architecture of the Linguistic– Spatial Interface. In: Bloom, P., Peterson, M.A., Nadal, L., Garrett, M.F. (eds.) Language and Space, pp. 1–30. MIT Press, Cambridge (1996)Google Scholar
  12. 12.
    De Vega, M., Cocude, M., Denis, M., Rodrigo, M.J., Zimmer, H.D.: The interface between language and visuo-spatial representations. In: Denis, M., Logie, R.H., Cornoldi, C., De Vega, M., Engelkamp, J. (eds.) Imagery, Language, and Visuo-Spatial Thinking, pp. 109–136. Psychology Press, Hove (2001)Google Scholar
  13. 13.
    Kemmerer, D.: ‘Near’ and ‘Far’ in Language and Perception. Cognition 73, 35–63 (1999)CrossRefGoogle Scholar
  14. 14.
    Emmorey, K.: The Effects of Modality on Spatial Language: How Signers and Speakers Talk about Space. In: Meier, R.P., Cormier, K., Quinto-Pozos, D. (eds.) Modality and Structure in Signed and Spoken Language, pp. 405–421. Cambridge University Press, Cambridge (2002)CrossRefGoogle Scholar
  15. 15.
    Agrafiotis, D., Canagarajah, N., Bull, D.R., Dye, M.: Perceptually optimised sign language video coding based on eye tracking analysis. Electron. Lett. 39(24), 1703–1705 (2003)CrossRefGoogle Scholar
  16. 16.
    Emmorey, K., Corina, D., Bellugi, U.: Differential processing of topographic and referential functions of space. In: Emmorey, K., Reilly, J. (eds.) Language, Gesture and Space, pp. 43–62. Lawrence Erlbaum Associates, Hillsdale (1995)Google Scholar
  17. 17.
    Emmorey, K.: Language, cognition, and the brain: Insights from sign language research. Lawrence Erlbaum Associates, Hillsdale (2001)Google Scholar
  18. 18.
    Engberg-Pedersen, E.: Space in Danish Sign Language: the Semantics and Morphosyntax of the Use of Space in a Visual Language. Signum Press, Hamburg (1993)Google Scholar
  19. 19.
    Supalla, T.: The Classifier System in ASL. In: Craig, C. (ed.) Noun Classification and Categorization, pp. 181–214. John Benjamins, Amsterdam (1986)CrossRefGoogle Scholar
  20. 20.
    Campbell, R., MacSweeney, M., Waters, D.: Sign Language and the Brain: a Review. J. Deaf. Stud. Deaf Ed. 13(1), 3–20 (2008)CrossRefGoogle Scholar
  21. 21.
    MacSweeney, M., Woll, B., Campbell, R., McGuire, P.K., David, A.S., Williams, S.C.R., Suckling, J., Calvert, G.A., Brammer, M.J.: Neural Systems Underlying British Sign Language and Audiovisual English Processing in Native Users. Brain 125, 1583–1593 (2002)CrossRefGoogle Scholar
  22. 22.
    Calvert, G.A., Bullmore, E.T., Brammer, M.J., Campbell, R., Williams, S.C., McGuire, P.K., Woodruff, P.W., Iversen, S.D., David, A.S.: Activation of Auditory Cortex During Silent Lipreading. Science 25:276 (5312), 593–596 (1997)CrossRefGoogle Scholar
  23. 23.
    MacSweeney, M., Amaro, E., Calvert, G., Campbell, R., David, A.S., McGuire, P.K., Williams, S.C., Woll, B., Brammer, M.J.: Silent Speechreading in the Absence of Scanner Noise: An Event-related fMRI Study. Neurorep. 11(8), 1729–1733 (2000)CrossRefGoogle Scholar
  24. 24.
    MacSweeney, M., Campbell, R., Calvert, G.A., McGuire, P.K., David, A.S., Suckling, J., Andrew, C., Woll, B., Brammer, M.J.: Dispersed Activation in the Left Temporal Cortex for Speech-reading in Congenitally Deaf People. Proc. Roy. Soc. B 268, 451–457 (2001)CrossRefGoogle Scholar
  25. 25.
    Haist, F., Song, A.W., Wild, K., Faber, T.L., Popp, C.A., Morris, R.D.: Linking Sight and Sound: fMRI Evidence of Primary Auditory Cortex Activation during Visual Word Recognition. Br Lang. 763, 340–350 (2001)CrossRefGoogle Scholar
  26. 26.
    Levänen, S., Jousmäki, V., Hari, R.: Vibration-induced Auditory-Cortex Activation in a Congenitally Deaf Adult. Curr. Biol. 8, 869–872 (1998)CrossRefGoogle Scholar
  27. 27.
    Nishimura, H., Hashikawa, K., Doi, K., Iwaki, T., Watanabe, Y., Kusuoka, H., Nishimura, T., Kubo, T.: Sign Language ’Heard’ in The Auditory Cortex. Nature 3976715, 116 (1999)CrossRefGoogle Scholar
  28. 28.
    Petitto, L.A., Zatorre, R.J., Gauna, K., Nikelski, E.J., Dostie, D., Evans, A.C.: Speech-like Cerebral Activity in Profoundly Deaf People Processing Signed Languages: Implications for the Neural Basis of Human Language. Proc. Nat. Acad. Sci. USA 97, 13961–13966 (2000)CrossRefGoogle Scholar
  29. 29.
    MacSweeney, M., Woll, B., Campbell, R., Calvert, G.A., McGuire, P.K., David, A.S., Simmons, A., Brammer, M.J.: Neural Correlates of British Sign Language Comprehension: Spatial Processing Demands of Topographic Language. J. Cog. Neurosci. 14, 1064–1075 (2002)CrossRefGoogle Scholar
  30. 30.
    Poizner, H., Klima, E., Bellugi, U.: What the Hands Reveal about the Brain. MIT Press, Cambridge (1987)Google Scholar
  31. 31.
    Hickok, G., Say, K., Bellugi, U., Klima, E.: The Basis of Hemispheric Asymmetries for Language and Spatial Cognition: Clues from Focal Brain Damage in 2 Deaf Native Signers. Aphasiol. 10, 577–591 (1996)CrossRefGoogle Scholar
  32. 32.
    Hickok, G., Bellugi, U., Klima, E.: The Neural Organization of Sign Language: Evidence from Sign Language Aphasia. Trans. Cog. Sci. 2, 129–136 (1998)CrossRefGoogle Scholar
  33. 33.
    Corina, D.P.: Aphasia in Users of Signed Languages. In: Coppens, P., Lebrun, Y., Basso, A. (eds.) Aphasia in Atypical Populations, pp. 261–309. Lawrence Erlbaum Associates, Mahwah (1998)Google Scholar
  34. 34.
    Corina, D.P.: The Processing of Sign Language: Evidence from Aphasia. In: Stemmer, B., Whitaker, H.A. (eds.) Handbook of Neurolinguistics, pp. 313–329. Academic Press, NY (1998)CrossRefGoogle Scholar
  35. 35.
    Atkinson, J.R., Campbell, R., Marshall, J., Thacker, A., Woll, B.: Understanding ‘not’: Neuropsychological Dissociations between Hand and Head Markers of Negation in BSL. Neuropsychologia 42, 214–229 (2004)CrossRefGoogle Scholar
  36. 36.
    Atkinson, J.R., Marshall, J., Woll, B., Thacker, A.: Testing Comprehension Abilities in Users of British Sign Language following CVA. Br Lang. 94(2), 233–248 (2005)CrossRefGoogle Scholar
  37. 37.
    Marshall, J., Atkinson, J.R., Smulovitch, E., Thacker, A., Woll, B.: Aphasia in a user of British Sign Language: Dissociation between sign and gesture. Cog. Neuropsychol. 21(5), 537–554 (2004)CrossRefGoogle Scholar
  38. 38.
    Marshall, J., Atkinson, J.R., Woll, B., Thacker, A.: Aphasia in a Bilingual User of British Sign Language and English: Effects of Cross Linguistic Cues. J. Cog. Neuropsychol. 22(6), 719–736 (2005)CrossRefGoogle Scholar
  39. 39.
    Corina, D.P., Poizner, H., Bellugi, U., Feinberg, T., Dowd, D., O’Grady-Batch, L.: Dissociation between Linguistic and Nonlinguistic Gestural Systems: a Case for Compositionality. Br Lang. 43, 414–447 (1992)CrossRefGoogle Scholar
  40. 40.
    Pizzuto, E., Volterra, V.: Iconicity and Transparency in Sign Languages: A Cross-linguistic Cross-cultural view. In: Emmorey, K., Harlan, L. (eds.) The Signs of Language Revisited: An Anthology in Honor of Ursula Bellugi and Edward Klima, pp. 261–286. Lawrence Erlbaum Associates, Hillsdale (2000)Google Scholar
  41. 41.
    Klima, E.S., Bellugi, U.: The Signs of Language. MIT Press, Cambridge (1979)Google Scholar
  42. 42.
    Emmorey, K., Grabowski, T., McCullough, S., Damasio, H., Ponto, L., Hichwa, R., Bellugi, U.: Motor-iconicity of Sign Language Does Not Alter the Neural Systems Underlying Tool and Action Naming. Br Lang. 89, 27–37 (2004)CrossRefGoogle Scholar
  43. 43.
    Neville, H., Bavelier, D., Corina, D., Rauschecker, J., Karni, A., Lalwani, A., Braun, A., Clark, V., Jezzard, P., Turner, R.: Cerebral Organization for Language in Deaf and Hearing Subjects: Biological Constraints and Effects of Experience. Proc. Nat. Acad. Sci. USA 95, 922–929 (1998)CrossRefGoogle Scholar
  44. 44.
    Newman, A.J., Bavelier, D., Corina, D., Jezzard, P., Neville, H.J.: A Critical Period for Right Hemisphere Recruitment in American Sign Language Processing. Nat. Neurosci. 5, 76–80 (2002)CrossRefGoogle Scholar
  45. 45.
    Paulesu, E., Mehler, J.: Right on in Sign Language. Nature 392, 233–234 (1998)CrossRefGoogle Scholar
  46. 46.
    Emmorey, K., Damasio, H., McCullough, S., Grabowski, T., Ponto, L., Hichwa, R., et al.: Neural Systems Underlying Spatial Language in American Sign Language. Neuroimage 17, 812–824 (2002)CrossRefGoogle Scholar
  47. 47.
    Kegl, J., Poizner, H.: Crosslinguistic/crossmodal Syntactic Consequences of Left-Hemisphere Damage: Evidence from an Aphasic Signer and his Identical Twin. Aphasiol. 11, 1–37 (1997)CrossRefGoogle Scholar
  48. 48.
    Loew, R.C., Kegl, J.A., Poizner, H.: Fractionation of the Components of Role Play in a Right-Hemisphere Lesioned Signer. Aphasiol. 11, 263–281 (1997)CrossRefGoogle Scholar
  49. 49.
    Hickok, G., Wilson, M., Clark, K., Klima, E.S., Kritchevsky, M., Bellugi, U.: Discourse Deficits Following Right Hemisphere Damage in Deaf Signers. Br Lang. 66, 233–248 (1999)CrossRefGoogle Scholar
  50. 50.
    Wapner, W., Hamby, S., Gardner, H.: The Role of the Right Hemisphere in the Apprehension of Complex Linguistic Materials. Br Lang. 14, 15–33 (1981)CrossRefGoogle Scholar
  51. 51.
    Kaplan, J.A., Brownell, H.R., Jacobs, J.R., Gardner, H.: The Effects of Right Hemisphere Damage on the Pragmatic Interpretation of Conversational Remarks. Br Lang. 38(2), 315–333 (1990)CrossRefGoogle Scholar
  52. 52.
    Rönnberg, J., Söderfeldt, B., Risberg, J.: The Cognitive Neuroscience of Signed Language. Acta. Psychol. 105(2-3), 237–254 (2000)CrossRefGoogle Scholar
  53. 53.
    Sutton-Spence, R., Day, L.: Mouthings and Mouth Gestures in British Sign Language. In: Boyes-Braem, P., Sutton-Spence, R. (eds.) The Hands are the Head of the Mouth, pp. 69–86. Signum Press, Hamburg (2001)Google Scholar
  54. 54.
    Boyes-Braem, P., Sutton-Spence, R. (eds.): The Hands are the Head of the Mouth. Signum Press, Hamburg (2002)Google Scholar
  55. 55.
    Waters, D., Campbell, R., Capek, C.M., Woll, B., David, A.S., McGuire, P.K., Brammer, M.J., MacSweeney, M.: Fingerspelling, Signed Language, Text and Picture Processing in Deaf Native Signers: The Role of the Mid-fusiform Gyrus. Neuroimage 35(3), 1287–1302 (2007)CrossRefGoogle Scholar
  56. 56.
    Capek, C., Woll, B., MacSweeney, M., Waters, D., David, A.S., McGuire, P.K., Brammer, M.J., Campbell, R.: Hand and Mouth: Cortical Correlates of Lexical Processing in BSL and Speechreading. J. Cog. Neurosci. 20, 1220–1234 (2008)CrossRefGoogle Scholar
  57. 57.
    McCullough, S., Emmorey, K., Sereno, M.: Neural Organization for Recognition of Grammatical Facial Expressions in Deaf ASL Signers and Hearing Nonsigners. Cog. Br. Res. 22, 192–203 (2005)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Bencie Woll
    • 1
  1. 1.Deafness Cognition and Language Research CentreUCLLondonUK

Personalised recommendations