Differential Prefrontal Response during Natural and Synthetic Speech Perception: An fNIR Based Neuroergonomics Study

  • Hasan Ayaz
  • Paul Crawford
  • Adrian Curtin
  • Mashaal Syed
  • Banu Onaral
  • Willem M. Beltman
  • Patricia A. Shewokis
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8027)


Synthetic speech has a growing role in human computer interaction and automated systems with the emergence of ubiquitous computing such as smart phones, car multimedia control and navigation systems. Cognitive processing costs associated with comprehension of synthetic speech relative to comprehension of natural speech have been demonstrated with behavioral (reaction time, accuracy, etc.) and self-reported (ratings, etc.) measures. In this neuroergonomics study, we have used optical brain imaging (fNIR: functional near infrared spectroscopy) to capture the brain activation of participants while they were listening to speech with varied quality, as well as natural speech. Results indicated a differential hemodynamic response with speech quality. As fNIR systems are safe, portable and record brain activation in real world settings, fNIR is a practical and minimally intrusive assessment tool for user experience researchers and can provide an objective metric for the design and development of next generation synthetic speech systems.


Optical Brain Imaging functional near infrared spectroscopy fNIR synthetic speech perception auditory processing 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Sharp, D.J.: Monitoring and the Controlled Processing of Meaning: Distinct Prefrontal Systems. Cerebral Cortex 14, 1–10 (2004)MathSciNetCrossRefGoogle Scholar
  2. 2.
    Hardee, J.B., Mayhorn, C.B.: Reexamining Synthetic Speech: Intelligibility and the Effects of Age, Task, and Speech Type on Recall. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 51, 1143–1147 (2007)CrossRefGoogle Scholar
  3. 3.
    Paris, C.R., Thomas, M.H., Gilson, R.D., Kincaid, J.P.: Linguistic Cues and Memory for Synthetic and Natural Speech. Human Factors: The Journal of the Human Factors and Ergonomics Society 42, 421–431 (2000)CrossRefGoogle Scholar
  4. 4.
    Benson, R.R., Whalen, D.H., Richardson, M., Swainson, B., Clark, V.P., Lai, S., Liberman, A.M.: Parametrically dissociating speech and nonspeech perception in the brain using fMRI. Brain Lang. 78, 364–396 (2001)CrossRefGoogle Scholar
  5. 5.
    Ayaz, H., Shewokis, P.A., Bunce, S., Izzetoglu, K., Willems, B., Onaral, B.: Optical brain monitoring for operator training and mental workload assessment. Neuroimage 59, 36–47 (2012)CrossRefGoogle Scholar
  6. 6.
    Girouard, A., Solovey, E.T., Jacob, R.J.K.: Designing a passive brain computer interface using real time classification of functional near-infrared spectroscopy. International Journal of Autonomous and Adaptive Communications Systems 6, 26–44 (2013)CrossRefGoogle Scholar
  7. 7.
    James, D.R.C., Orihuela-Espina, F., Leff, D.R., Sodergren, M.H., Athanasiou, T., Darzi, A.W., Yang, G.Z.: The ergonomics of natural orifice translumenal endoscopic surgery (NOTES) navigation in terms of performance, stress, and cognitive behavior. Surgery 149, 525–533 (2011)CrossRefGoogle Scholar
  8. 8.
    Ayaz, H., Shewokis, P.A., İzzetoğlu, M., Çakır, M.P., Onaral, B.: Tangram solved? Prefrontal cortex activation analysis during geometric problem solving. In: 34th Annual International IEEE EMBS Conference, pp. 4724–4727. IEEE (2012)Google Scholar
  9. 9.
    Çiftçi, K., Sankur, B., Kahya, Y.P., Akın, A.: Functional Clusters in the Prefrontal Cortex during Mental Arithmetic. In: 16th European Signal Processing Conference (EUSIPCO 2008), Lausanne, Switzerland, pp. 1–4 (2008)Google Scholar
  10. 10.
    Hampshire, A., Thompson, R., Duncan, J., Owen, A.M.: Lateral prefrontal cortex subregions make dissociable contributions during fluid reasoning. Cerebral Cortex 21, 1–10 (2011)CrossRefGoogle Scholar
  11. 11.
    Ayaz, H., Bunce, S., Shewokis, P., Izzetoglu, K., Willems, B., Onaral, B.: Using Brain Activity to Predict Task Performance and Operator Efficiency. In: Zhang, H., Hussain, A., Liu, D., Wang, Z. (eds.) BICS 2012. LNCS, vol. 7366, pp. 147–155. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  12. 12.
    Power, S.D., Kushki, A., Chau, T.: Towards a system-paced near-infrared spectroscopy brain-computer interface: differentiating prefrontal activity due to mental arithmetic and mental singing from the no-control state. Journal of Neural Engineering 8, 066004 (2011)Google Scholar
  13. 13.
    Ayaz, H., Cakir, M.P., Izzetoglu, K., Curtin, A., Shewokis, P.A., Bunce, S.C., Onaral, B.: Monitoring expertise development during simulated UAV piloting tasks using optical brain imaging. In: Aerospace Conference, 2012 IEEE, pp. 1–11 (2012)Google Scholar
  14. 14.
    Shewokis, P.A., Ayaz, H., Izzetoglu, M., Bunce, S., Gentili, R.J., Sela, I., Izzetoglu, K., Onaral, B.: Brain in the Loop: Assessing Learning Using fNIR in Cognitive and Motor Tasks. In: Schmorrow, D.D., Fidopiastis, C.M. (eds.) FAC 2011. LNCS, vol. 6780, pp. 240–249. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  15. 15.
    Pfurtscheller, G., Bauernfeind, G., Wriessnegger, S.C., Neuper, C.: Focal frontal (de)oxyhemoglobin responses during simple arithmetic. Int. J. Psychophysiol. 76, 186–192 (2010)CrossRefGoogle Scholar
  16. 16.
    Parasuraman, R.: Neuroergonomics Brain, Cognition, and Performance at Work. Current Directions in Psychological Science 20, 181–186 (2011)CrossRefGoogle Scholar
  17. 17.
    Dieler, A.C., Tupak, S.V., Fallgatter, A.J.: Functional near-infrared spectroscopy for the assessment of speech related tasks. Brain Lang. 121, 90–109 (2012)CrossRefGoogle Scholar
  18. 18.
    Oldfield, R.C.: The assessment and analysis of handedness: the Edinburgh inventory. Neuropsychologia 9, 97–113 (1971)CrossRefGoogle Scholar
  19. 19.
    ITU-T: Recommendation P.835 Subjective test methodology for evaluating speech communication systems that include noise suppression algorithm, vol. (11/2003). International Telecommunication Union, Geneva (2003) Google Scholar
  20. 20.
    Ayaz, H., Izzetoglu, M., Platek, S.M., Bunce, S., Izzetoglu, K., Pourrezaei, K., Onaral, B.: Registering fNIR data to brain surface image using MRI templates. In; Conf. Proc. IEEE Eng. Med. Biol. Soc., pp. 2671–2674 (2006)Google Scholar
  21. 21.
    Ayaz, H., Shewokis, P.A., Curtin, A., Izzetoglu, M., Izzetoglu, K., Onaral, B.: Using MazeSuite and Functional Near Infrared Spectroscopy to Study Learning in Spatial Navigation. J. Vis. Exp., e3443 (2011)Google Scholar
  22. 22.
    Ayaz, H., Izzetoglu, M., Shewokis, P.A., Onaral, B.: Sliding-window Motion Artifact Rejection for Functional Near-Infrared Spectroscopy. In: Conf. Proc. IEEE Eng. Med. Biol. Soc., pp. 6567–6570 (2010)Google Scholar
  23. 23.
    Benjamini, Y., Hochberg, Y.: Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society. Series B (Methodological) 57, 289–300 (1995)MathSciNetzbMATHGoogle Scholar
  24. 24.
    Singh, A.K., Dan, I.: Exploring the false discovery rate in multichannel NIRS. Neuroimage 33, 542–549 (2006)CrossRefGoogle Scholar
  25. 25.
    Paas, F.G.W.C., Van Merriënboer, J.J.G.: The efficiency of instructional conditions: An approach to combine mental effort and performance measures. Human Factors: The Journal of the Human Factors and Ergonomics Society 35, 737–743 (1993)Google Scholar
  26. 26.
    Husain, F.T., Fromm, S.J., Pursley, R.H., Hosey, L.A., Braun, A.R., Horwitz, B.: Neural bases of categorization of simple speech and nonspeech sounds. Human Brain Mapping 27, 636–651 (2006)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Hasan Ayaz
    • 1
    • 2
  • Paul Crawford
    • 3
  • Adrian Curtin
    • 1
    • 2
  • Mashaal Syed
    • 1
    • 2
  • Banu Onaral
    • 1
    • 2
  • Willem M. Beltman
    • 3
  • Patricia A. Shewokis
    • 1
    • 2
    • 4
  1. 1.School of Biomedical Engineering, Science & Health SystemsDrexel UniversityPhiladelphiaUSA
  2. 2.Cognitive Neuroengineering and Quantitative Experimental Research (CONQUER) CollaborativeDrexel UniversityPhiladelphiaUSA
  3. 3.Intel LabsIntel CorporationSanta ClaraUSA
  4. 4.Nutrition Sciences Department, College of Nursing and Health ProfessionsDrexel UniversityPhiladelphiaUSA

Personalised recommendations