Skip to main content
SpringerLink
Log in

The effects of visual material and temporal synchrony on the processing of letters and speech sounds

  • Research Article
  • Published:
Experimental Brain Research Aims and scope Submit manuscript

Abstract

Associating letters with speech sounds is essential for reading skill acquisition. In the current study, we aimed at determining the effects of different types of visual material and temporal synchrony on the integration of letters and speech sounds. To this end, we recorded the mismatch negativity (MMN), an index of automatic change detection in the brain, from literate adults. Subjects were presented with auditory consonant–vowel syllable stimuli together with visual stimuli, which were either written syllables or scrambled pictures of the written syllables. The visual stimuli were presented in half of the blocks synchronously with the auditory stimuli and in the other half 200 ms before the auditory stimuli. The auditory stimuli were consonant, vowel or vowel length changes, or changes in syllable frequency or intensity presented by using the multi-feature paradigm. Changes in the auditory stimuli elicited MMNs in all conditions. MMN amplitudes for the consonant and frequency changes were generally larger for the sounds presented with written syllables than with scrambled syllables. Time delay diminished the MMN amplitude for all deviants. The results suggest that speech sound processing is modulated when the sounds are presented with letters versus non-linguistic visual stimuli, and further, that the integration of letters and speech sounds seems to be dependent on precise temporal alignment. Moreover, the results indicate that with our paradigm, a variety of parameters relevant and irrelevant for reading can be tested within one experiment.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  • Alho K (1995) Cerebral generators of mismatch negativity (MMN) and its magnetic counterpart (MMNm) elicited by sound changes. Ear Hear 16(1):38–51

    Article  PubMed  CAS  Google Scholar 

  • Alho K, Connolly JF, Cheour M, Lehtokoski A, Huotilainen M, Virtanen J, Aulanko R, Ilmoniemi RJ (1998) Hemispheric lateralization in preattentive processing of speech sounds. Neurosci Lett 258(1):9–12

    Article  PubMed  CAS  Google Scholar 

  • Alku P, Tiitinen H, Näätänen R (1999) A method for generating natural-sounding speech stimuli for cognitive brain research. Clin Neurophysiol 110(8):1329–1333

    Article  PubMed  CAS  Google Scholar 

  • Besle J, Fort A, Giard MH (2005) Is the auditory sensory memory sensitive to visual information? Exp Brain Res 166(3–4):337–344

    Article  PubMed  Google Scholar 

  • Cheour M, Ceponiene R, Lehtokoski A, Luuk A, Allik J, Alho K, Näätänen R (1998) Development of language-specific phoneme representations in the infant brain. Nat Neurosci 1(5):351–353

    Article  PubMed  CAS  Google Scholar 

  • Colin C, Radeau M, Soquet A, Deltenre P (2004) Generalization of the generation of a MMN by illusory McGurk percepts: Voiceless consonants. Clin Neurophysiol 115(9):1989–2000

    Article  PubMed  CAS  Google Scholar 

  • Delorme A, Makeig S (2004) Eeglab: an open source toolbox for analysis of single-trial eeg dynamics including independent component analysis. J Neurosci Methods 134(1):9–21

    Article  PubMed  Google Scholar 

  • Frost R, Repp BH, Katz L (1988) Can speech perception be influenced by simultaneous presentation of print? J Memory Lang 27:741–755

    Article  Google Scholar 

  • Froyen D, van Atteveldt N, Bonte M, Blomert L (2008) Cross-modal enhancement of the MMN to speech-sounds indicates early and automatic integration of letters and speech-sounds. Neurosci Lett 430(1):23–28

    Article  PubMed  CAS  Google Scholar 

  • Froyen DJ, Bonte ML, van Atteveldt N, Blomert L (2009) The long road to automation: neurocognitive development of letter-speech sound processing. J Cogn Neurosci 21(3):567–580

    Article  PubMed  Google Scholar 

  • Giard MH, Lavikainen J, Reinikainen K, Perrin F, Bertrand O, Pernier J, Näätänen R (1995) Seperate representation of stimulus frequency, intensity, and duration in auditory sensory memory: an event-related potential and dipole-model analysis. J Cognit Neurosci 7:133–143

    Article  Google Scholar 

  • Goswami U (2002) Cross-linguistic aspects of literacy development and dyslexia. Ann Dyslexia 52:141–163

    Article  Google Scholar 

  • Jaramillo M, Ilvonen T, Kujala T, Alku P, Tervaniemi M, Alho K (2001) Are different kinds of acoustic features processed differently for speech and non-speech sounds? Brain Res Cogn Brain Res 12(3):459–466

    Article  PubMed  CAS  Google Scholar 

  • Kirmse U, Ylinen S, Tervaniemi M, Vainio M, Schröger E, Jacobsen T (2008) Modulation of the mismatch negativity (MMN) to vowel duration changes in native speakers of Finnish and German as a result of language experience. Int J Psychophysiol 67(2):131–143

    PubMed  Google Scholar 

  • Kujala T, Tervaniemi M, Schröger E (2007) The mismatch negativity in cognitive and clinical neuroscience: theoretical and methodological considerations. Biol Psychol 74(1):1–19

    Article  PubMed  Google Scholar 

  • Lovio R, Pakarinen S, Huotilainen M, Alku P, Silvennoinen S, Näätänen R, Kujala T (2009) Auditory discrimination profiles of speech sound changes in 6-year-old children as determined with the multi-feature MMN paradigm. Clin Neurophysiol 120(5):916–921

    Article  PubMed  Google Scholar 

  • Luck SJ (2005) An introduction to the event-related potential technique. The MIT Press, Cambridge

    Google Scholar 

  • Massaro DW, Cohen MC, Thompson LA (1988) Visible language in speech perception: lipreading and reading. Visible Lang 22:8–31

    Google Scholar 

  • McGurk H, MacDonald J (1976) Hearing lips and seeing voices. Nature 264(5588):746–748

    Article  PubMed  CAS  Google Scholar 

  • Muller-Gass A, Stelmack RM, Campbell (2006) The effect of visual task difficulty and attentional direction on the detection of acoustic change as indexed by the mismatch negativity. Brain Res 1078:112–130

    Article  PubMed  CAS  Google Scholar 

  • Näätänen R, Michie PT (1979) Early selective-attention effects on the evoked potential: a critical review and reinterpretation. Biol Psychol 8(2):81–136

    Article  PubMed  Google Scholar 

  • Näätänen R, Gaillard AW, Mäntysalo S (1978) Early selective-attention effect on evoked potential reinterpreted. Acta Psychol (Amst) 42(4):313–329

    Article  Google Scholar 

  • Näätänen R, Lehtokoski A, Lennes M, Cheour M, Huotilainen M, Iivonen A, Vainio M, Alku P, Ilmoniemi RJ, Luuk A, Allik J, Sinkkonen J, Alho K (1997) Language-specific phoneme representations revealed by electric and magnetic brain responses. Nature 385(6615):432–434

    Article  PubMed  Google Scholar 

  • Näätänen R, Pakarinen S, Rinne T, Takegata R (2004) The mismatch negativity (MMN): towards the optimal paradigm. Clin Neurophysiol 115(1):140–144

    Article  PubMed  Google Scholar 

  • Näätänen R, Paavilainen P, Rinne T, Alho K (2007) The mismatch negativity (MMN) in basic research of central auditory processing: a review. Clin Neurophysiol 118(12):2544–2590

    Article  PubMed  Google Scholar 

  • Novak GP, Ritter W, Vaughan HG Jr, Wiznitzer ML (1990) Differentiation of negative event-related potentials in an auditory discrimination task. Electroencephalogr Clin Neurophysiol 75(4):255–275

    Article  PubMed  CAS  Google Scholar 

  • Paavilainen P, Alho K, Reinikainen K, Sams M, Näätänen R (1991) Right hemisphere dominance of different mismatch negativities. Electroencephalogr Clin Neurophysiol 78(6):466–479

    Article  PubMed  CAS  Google Scholar 

  • Pakarinen S, Lovio R, Huotilainen M, Alku P, Näätänen R, Kujala T (2009) Fast multi-feature paradigm for recording several mismatch negativities (MMNs) to phonetic and acoustic changes in speech sounds. Biol Psychol 82(3):219–226

    Article  PubMed  Google Scholar 

  • Pulvermüller F, Kujala T, Shtyrov Y, Simola J, Tiitinen H, Alku P, Alho K, Martinkauppi S, Ilmoniemi RJ, Näätänen R (2001) Memory traces for words as revealed by the mismatch negativity. Neuroimage 14(3):607–616

    Article  PubMed  Google Scholar 

  • Rinne T, Alho K, Alku P, Holi M, Sinkkonen J, Virtanen J, Bertrand O, Näätänen R (1999) Analysis of speech sounds is left-hemisphere predominant at 100–150 ms after sound onset. NeuroReport 10:1113–1117

    Article  PubMed  CAS  Google Scholar 

  • Sams M, Aulanko R, Hämäläinen M, Hari R, Lounasmaa OV, Lu ST, Simola J (1991) Seeing speech: visual information from lip movements modifies activity in the human auditory cortex. Neurosci Lett 127(1):141–145

    Article  PubMed  CAS  Google Scholar 

  • Shtyrov Y, Kujala T, Ahveninen J, Tervaniemi M, Alku P, Ilmoniemi RJ, Näätänen R (1998) Background acoustic noise and the hemispheric lateralization of speech processing in the human brain: magnetic mismatch negativity study. Neurosci Lett 251(2):141–144

    Article  PubMed  CAS  Google Scholar 

  • Shtyrov Y, Kujala T, Palva S, Ilmoniemi RJ, Näätänen R (2000) Discrimination of speech and of complex nonspeech sounds of different temporal structure in the left and right cerebral hemispheres. Neuroimage 12(6):657–663

    Article  PubMed  CAS  Google Scholar 

  • Siegel LS, Faux D (1989) Acquisition of certain grapheme-phoneme correspondence in normally achieving and disabled readers. Read Writ : An Interdiscip J 1:37–52

    Google Scholar 

  • Snowling MJ (1981) Phonemic deficits in developmental dyslexia. Psychol Res 43(2):219–234

    Article  PubMed  CAS  Google Scholar 

  • Sorokin A, Alku P, Kujala T (2010) Change and novelty detection in speech and non-speech sound streams. Brain Res 1327:77–90

    Article  PubMed  CAS  Google Scholar 

  • Sussman E, Ritter W, Vaughan HG Jr (1998) Attention affects the organization of auditory input associated with the mismatch negativity system. Brain Res 789(1):130–138

    Article  PubMed  CAS  Google Scholar 

  • Swets JA (1964) Signal detection and recognition by human observers. Wiley, New York

    Google Scholar 

  • Takegata R, Tervaniemi M, Alku P, Ylinen S, Näätänen R (2008) Parameter-specific modulation of the mismatch negativity to duration decrement and increment: evidence for asymmetric processes. Clin Neurophysiol 119(7):1515–1523

    Article  PubMed  Google Scholar 

  • Tervaniemi M, Jacobsen T, Rottger S, Kujala T, Widmann A, Vainio M, Näätänen R, Schröger E (2006) Selective tuning of cortical sound-feature processing by language experience. Eur J Neurosci 23(9):2538–2541

    Article  PubMed  CAS  Google Scholar 

  • Vainio M, Järvikivi J (2007) Focus in production: tonal shape, intensity and word order. J Acoust Soc Am 121 (2):EL55-61

    Google Scholar 

  • van Atteveldt NM, Formisano E, Blomert L, Goebel R (2007) The effect of temporal asynchrony on the multisensory integration of letters and speech sounds. Cereb Cortex 17(4):962–974

    Article  PubMed  Google Scholar 

  • Widmann A, Kujala T, Tervaniemi M, Kujala A, Schröger E (2004) From symbols to sounds: visual symbolic information activates sound representations. Psychophysiology 41(5):709–715

    Article  PubMed  Google Scholar 

  • Widmann A, Gruber T, Kujala T, Tervaniemi M, Schröger E (2007) Binding symbols and sounds: evidence from event-related oscillatory gamma-band activity. Cereb Cortex 17(11):2696–2702

    Article  PubMed  Google Scholar 

  • Winkler I, Kujala T, Tiitinen H, Sivonen P, Alku P, Lehtokoski A, Czigler I, Csepe V, Ilmoniemi RJ, Näätänen R (1999) Brain responses reveal the learning of foreign language phonemes. Psychophysiology 36(5):638–642

    Article  PubMed  CAS  Google Scholar 

  • Woldorff MG, Hackley SA, Hillyard SA (1991) The effects of channel-selective attention on the mismatch negativity wave elicited by deviant tones. Psychophysiology 28(1):30–42

    Article  PubMed  CAS  Google Scholar 

  • Woldorff MG, Hillyard SA, Gallen CC, Hampson SR, Bloom FE (1998) Magnetoencephalographic recordings demonstrate attentional modulation of mismatch-related neural activity in human auditory cortex. Psychophysiology 35(3):283–292

    Article  PubMed  CAS  Google Scholar 

  • Yanz JL (1984) The application of the theory of signal detection to the assessment of speech perception. Ear Hear 5:64–71

    Article  PubMed  CAS  Google Scholar 

  • Ylinen S, Shestakova A, Alku P, Huotilainen M (2005) The perception of phonological quantity based on durational cues by native speakers, second-language users and nonspeakers of Finnish. Lang Speech 48(Pt 3):313–338

    Article  PubMed  Google Scholar 

  • Yucel G, Petty C, McCarthy G, Belger A (2005) Graded visual attention modulates brain responses evoked by task-irrelevant auditory pitch changes. J Cogn Neurosci 17(12):1819–1828

    Article  PubMed  Google Scholar 

Download references

Acknowledgments

The present study was financially supported by the Centre for International Mobility (CIMO, Helsinki, Finland), the Academy of Finland (grant 122745), and the University of Helsinki. Special thanks to Prof. Paavo Alku for creating the auditory stimuli, and Mr. Michael Ranft for his valuable contribution in data analyses.

Author information

Authors and Affiliations

  1. Cognitive Brain Research Unit, Institute of Behavioural Sciences, University of Helsinki, Helsinki, Finland

    Maria Mittag, Rika Takegata & Teija Kujala

Authors
  1. Maria Mittag
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rika Takegata
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Teija Kujala
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Maria Mittag.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Mittag, M., Takegata, R. & Kujala, T. The effects of visual material and temporal synchrony on the processing of letters and speech sounds. Exp Brain Res 211, 287–298 (2011). https://doi.org/10.1007/s00221-011-2686-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00221-011-2686-z

Keywords

Search

Navigation