Toward an Open Data Repository and Meta-Analysis of Cognitive Data Using fNIRS Studies of Emotion

  • Sarah BrattEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10284)


HCI research has increasingly incorporated the use of neurophysiological sensors to identify users’ cognitive and affective states. However, a persistent problem in machine learning on cognitive data is generalizability across participants. A proposed solution has been aggregating cognitive and survey data across studies to generate higher sample populations for machine learning and statistical analyses to converge in stable, generalizable results. In this paper, I argue that large data-sharing projects can facilitate the aggregation of results of brain imaging studies to address these issues, by smoothing noise in high-dimensional datasets. This paper contributes a small step towards large cognitive data sharing systems-design by proposing methods that facilitate the merging of currently incompatible fNIRS and FMRI datasets through term-based metadata analysis. To that end, I analyze 20 fNIRS studies of emotion using content analysis for: (1) synonym terms and definitions for ‘emotion,’ (2) the experimental stimuli, and (3) the use or non-use of self-report surveys. Results suggest that fNIRS studies of emotion have stable synonymy, using technical and folk conceptualizations of affective terms within and between publications to refer to emotion. The studies use different stimuli to elicit emotion but also show commonalities between shared use of standardized stimuli materials and self-report surveys. These similarities in conceptual synonymy and standardized experiment materials indicate promise for neuroimaging communities to establish open-data repositories based on metadata term-based analyses. This work contributes to efforts toward merging datasets across studies and between labs, unifying new modalities in neuroimaging such as fNIRS with fMRI datasets, increasing generalizability of machine learning models, and promoting the acceleration of science through open data-sharing infrastructure.


Data repository Cognitive data fNIRS Emotion Metadata Machine learning HCI 



The author thanks the Syracuse University Media Interface Network Design (M.I.N.D.) Lab at S.I. Newhouse School of Public Communications and colleagues for support.


  1. Abdul Serwadda, V.P.: fNIRS: A New Modality for Brain Activity-Based Biometric Authentication (2015)Google Scholar
  2. Adolphs, R., Damasio, H., Tranel, D., Damasio, A.R.: Cortical systems for the recognition of emotion in facial expressions. J. Neurosci. 16(23), 7678–7687 (1996)Google Scholar
  3. Association, W.M., et al.: World medical association declaration of Helsinki. Ethical principles for medical research involving human subjects. Nurs. Ethics 9(1), 105 (2002)CrossRefGoogle Scholar
  4. Balconi, M., Grippa, E., Vanutelli, M.E.: Resting lateralized activity predicts the cortical response and appraisal of emotions: an fNIRS study. Soc. Cogn. Affect. Neurosci. 10(12), 1607–1614 (2015). doi:
  5. Banks, S.J., Eddy, K.T., Angstadt, M., Nathan, P.J., Phan, K.L.: Amygdala-frontal connectivity during emotion regulation. Soc. Cogn. Affect. Neurosci. 2(4), 303–312 (2007). doi:
  6. Bigliassi, M., Barreto-Silva, V., Altimari, L.R., Vandoni, M., Codrons, E., Buzzachera, C.F.: How motivational and calm music may affect the prefrontal cortex area and emotional responses: a functional near-infrared spectroscopy (fNIRS) study. Percept. Mot. Skills 120(1), 202–218 (2015). doi:
  7. Blair, R.J.R., Morris, J.S., Frith, C.D., Perrett, D.I., Dolan, R.J.: Dissociable neural responses to facial expressions of sadness and anger. Brain 122(5), 883–893 (1999)CrossRefGoogle Scholar
  8. Blood, A.J., Zatorre, R.J., Bermudez, P., Evans, A.C.: Emotional responses to pleasant and unpleasant music correlate with activity in paralimbic brain regions. Nat. Neurosci. 2(4), 382–387 (1999). doi:
  9. Bradley, M., Lang, P.J.: Measuring emotion: the self-assessment manikin and the semantic differential. J. Behav. Ther. Exp. Psychiatry 25(1), 49–59 (1994)CrossRefGoogle Scholar
  10. Breiter, H.C., Etcoff, N.L., Whalen, P.J., Kennedy, W.A., Rauch, S.L., Buckner, R.L., Rosen, B.R.: Response and habituation of the human amygdala during visual processing of facial expression. Neuron 17(5), 875–887 (1996)CrossRefGoogle Scholar
  11. Brink, T.T., Urton, K., Held, D., Kirilina, E., Hofmann, M.J., Klann-Delius, G., Kuchinke, L.: The role of orbitofrontal cortex in processing empathy stories in 4- to 8-year-old children. Front. Psychol. 2, (2011). doi:
  12. Bunce, S., Izzetoglu, M., Izzetoglu, K., Onaral, B., Pourrezaei, K.: Functional near infrared spectroscopy: an emerging neuroimaging modality. IEEE Eng. Med. Biol. Mag. Special Issue Clin. Neuroeng. 25(4), 54–62 (2006)CrossRefGoogle Scholar
  13. Codispoti, M., Surcinelli, P., Baldaro, B.: Watching emotional movies: affective reactions and gender differences. Int. J. Psychophysiol.: Official J. Int. Organ. Psychophysiol. 69(2), 90–95 (2008). doi:
  14. Costa, M., Bratt, S.: Truthiness: challenges associated with employing machine learning on neurophysiological sensor data. In: Schmorrow, D.D., Fidopiastis, C.M. (eds.) AC 2016. LNCS (LNAI), vol. 9743, pp. 159–164. Springer, Cham (2016). doi: 10.1007/978-3-319-39955-3_15 Google Scholar
  15. Diemer, J., Alpers, G.W., Peperkorn, H.M., Shiban, Y., Mühlberger, A.: The impact of perception and presence on emotional reactions: a review of research in virtual reality. Frontiers Psychol. 6 (2015). doi:
  16. Edwards, A.: The relationship between the judged desirability of a trait and the probability that the trait will be endorsed. J. Appl. Psychol. 37, 90–93 (1953)CrossRefGoogle Scholar
  17. Ekman, P.: Facial expression and emotion. Am. Psychol. 48(4), 384 (1993)CrossRefGoogle Scholar
  18. Ekman, P., Friesen, W.V.: Constants across cultures in the face and emotion. J. Pers. Soc. Psychol. 17(2), 124–129 (1971). doi:
  19. Elo, S., Kyngäs, H.: The qualitative content analysis process. J. Adv. Nurs. 62(1), 107–115 (2008)CrossRefGoogle Scholar
  20. Ernst, L.H., Plichta, M.M., Lutz, E., Zesewitz, A.K., Tupak, S.V., Dresler, T., Fallgatter, A.J.: Prefrontal activation patterns of automatic and regulated approach-avoidance reactions - a functional near-infrared spectroscopy (fNIRS) study. Cortex; J. Devoted Study Nerv. Syst. Behav. 49(1), 131–142 (2013). doi:
  21. Glotzbach, E., Mühlberger, A., Gschwendtner, K., Fallgatter, A.J., Pauli, P., Herrmann, M.J.: Prefrontal brain activation during emotional processing: a functional near infrared spectroscopy study (fNIRS). Open Neuroimaging J. 5, 33–39 (2011). doi:
  22. Gorini, A., Mosso, J.L., Mosso, D., Pineda, E., Ruíz, N.L., Ramíez, M., Riva, G.: Emotional response to virtual reality exposure across different cultures: the role of the attribution process. Cyberpsychol. Behav.: Impact Internet, Multimedia Virtual Real. Behav. Soc. 12(6), 699–705 (2009). doi:
  23. Grimes, D., Tan, D.S., Hudson, S.E., Shenoy, P., Rao, R.P.N.: Feasibility and pragmatics of classifying working memory load with an electroencephalograph. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 835–844. ACM, New York (2008). doi:
  24. Gygax, L., Reefmann, N., Wolf, M., Langbein, J.: Prefrontal cortex activity, sympatho-vagal reaction and behaviour distinguish between situations of feed reward and frustration in dwarf goats. Behav. Brain Res. 239, 104–114 (2013). doi:
  25. Hengehold, L.: “My body, this paper, this fire”: the fate of emotion in Foucault’s Kantian legacy. Philos. Today 47, 45–55 (2003)CrossRefGoogle Scholar
  26. Hosseini, S.M., Mano, H., Rostami, Y., Takahashi, M., Sugiura, M., Kawashima, R.: Decoding what one likes or dislikes from single-trial fNIRS measurements. Neuroreport, 22(6), 269–273 (2011). doi:
  27. Hudson, M.B., Nicolas, S.C., Howser, M.E., Lipsett, K.E., Robinson, I.W., Pope, L.J., Friedman, D.R.: Examining how gender and emoticons influence facebook jealousy. Cyberpsychol. Behav. Soc. Network. 18(2), 87–92 (2015). doi:
  28. Iacucci, G., Kuutti, K.: Everyday life as a stage in creating and performing scenarios for wireless devices. Pers. Ubiquitous Comput. 6(4), 299–306 (2002). doi:
  29. El Kaliouby, R., Picard, R., Baron-Cohen, S.: Affective computing and autism. Ann. New York Acad. Sci. 1093(1), 228–248 (2006). doi:
  30. Kassam, K.S., Markey, A.R., Cherkassky, V.L., Loewenstein, G., Just, M.A.: Identifying emotions on the basis of neural activation. PLoS ONE 8(6), e66032 (2013)CrossRefGoogle Scholar
  31. Kassam, K.S., Mendes, W.B.: The effects of measuring emotion: physiological reactions to emotional situations depend on whether someone is asking. PLOS ONE, 8(6), e64959 (2013). doi:
  32. Kopf, J., Dresler, T., Reicherts, P., Herrmann, M.J., Reif, A.: The effect of emotional content on brain activation and the late positive potential in a word n-back task. PLOS ONE, 8(9), e75598 (2013). doi:
  33. Kreplin, U., Fairclough, S.H.: Activation of the rostromedial prefrontal cortex during the experience of positive emotion in the context of esthetic experience. an fNIRS study. Front. Hum. Neurosci. 7 (2013). doi:
  34. Leon-Carrion, J., Damas, J., Izzetoglu, K., Pourrezai, K., Martín-Rodríguez, J.F., Barroso y Martin, J.M., Dominguez-Morales, M.R.: Differential time course and intensity of PFC activation for men and women in response to emotional stimuli: a functional near-infrared spectroscopy (fNIRS) study. Neurosci. Lett. 403(1–2), 90–95 (2006). doi:
  35. León-Carrión, J., Martín-Rodríguez, J.F., Damas-López, J., Pourrezai, K., Izzetoglu, K., y Martin, J.M.B., Domínguez-Morales, M.R.: A lasting post-stimulus activation on dorsolateral prefrontal cortex is produced when processing valence and arousal in visual affective stimuli. Neurosci. Lett. 422(3), 147–152 (2007). doi:
  36. Link, G.J., Germonprez, M., Goggins, S., Hemsley, J., Rand, B., Squire, M.: The open community data exchange: advancing data sharing and discovery in open online community science. In: Proceedings of the 12th International Symposium on Open Collaboration Companion, p. 7. ACM (2016).
  37. Neurosynth (n.d.). Accessed 20 Oct 2016.
  38. Nicholson, A.A., Rabellino, D., Densmore, M., Frewen, P.A., Paret, C., Kluetsch, R., et al.: The neurobiology of emotion regulation in posttraumatic stress disorder: amygdala downregulation via real-time fMRI neurofeedback. Hum. Brain Mapp. 38(1), 541–560 (2017)CrossRefGoogle Scholar
  39. Noah, J.A., Ono, Y., Nomoto, Y., Shimada, S., Tachibana, A., Zhang, X., Hirsch, J.: fMRI validation of fNIRS measurements during a naturalistic task. J. Vis. Exp. (100) (2015). doi:
  40. Perlman, S.B., Luna, B., Hein, T.C., Huppert, T.J.: fNIRS evidence of prefrontal regulation of frustration in early childhood. NeuroImage, 85 Pt 1, 326–334 (2014). doi:
  41. Phan, K.L., Wager, T., Taylor, S.F., Liberzon, I.: Functional neuroanatomy of emotion: a meta-analysis of emotion activation studies in PET and fMRI. Neuroimage 16(2), 331–348 (2002)CrossRefGoogle Scholar
  42. Rothbart, M.K., Ahadi, S.A., Hershey, K.L., Fisher, P.: Investigations of temperament at three to seven years: the children’s behavior questionnaire. Child Dev. 72(5), 1394–1408 (2001)CrossRefGoogle Scholar
  43. Russell, J.A., Mehrabian, A.: Evidence for a three-factor theory of emotions. J. Res. Pers. 11(3), 273–294 (1977)CrossRefGoogle Scholar
  44. Rutkowski, T.M., Tanaka, T., Cichocki, A., Erickson, D., Cao, J., Mandic, D.P.: Interactive component extraction from fEEG, fNIRS and peripheral biosignals for affective brain–machine interfacing paradigms. Comput. Hum. Behav. 27(5), 1512–1518 (2011)CrossRefGoogle Scholar
  45. Scherer, K.R.: What are emotions? And how can they be measured? Soc. Sci. Inf. 44(4), 695–729 (2005). doi:
  46. Schneider, S., Christensen, A., Häußinger, F.B., Fallgatter, A.J., Giese, M.A., Ehlis, A.-C.: Show me how you walk and i tell you how you feel — a functional near-infrared spectroscopy study on emotion perception based on human gait. NeuroImage, 85, Part 1, 380–390 (2014). doi:
  47. Sevick, E.M., Chance, B., Leigh, J., Nioka, S., Maris, M.: Quantitation of time-and frequency-resolved optical spectra for the determination of tissue oxygenation. Anal. Biochem. 195(2), 330–351 (1991)CrossRefGoogle Scholar
  48. Shimoda, N., Takeda, K., Imai, I., Kaneko, J., Kato, H.: Cerebral laterality differences in handedness: a mental rotation study with NIRS. Neurosci. Lett. 430(1), 43–47 (2008). doi:
  49. Soleymani, M., Asghari-Esfeden, S., Fu, Y., Pantic, M.: Analysis of EEG signals and facial expressions for continuous emotion detection. IEEE Trans. Affect. Comput. 7(1), 17–28 (2016)CrossRefGoogle Scholar
  50. Stevens, J.S., Hamann, S.: Sex differences in brain activation to emotional stimuli: a meta-analysis of neuroimaging studies. Neuropsychologia 50(7), 1578–1593 (2012). doi:
  51. Thayer, J.F., Åhs, F., Fredrikson, M., Sollers, J.J., Wager, T.D.: A meta-analysis of heart rate variability and neuroimaging studies: implications for heart rate variability as a marker of stress and health. Neurosci. Biobehav. Rev. 36(2), 747–756 (2012). doi:
  52. Thayer, J.F., Lane, R.D.: Claude Bernard and the heart–brain connection: further elaboration of a model of neurovisceral integration. Neurosci. Biobehav. Rev. 33(2), 81–88 (2009)CrossRefGoogle Scholar
  53. Thomas, C., Altenmüller, E., Marckmann, G., Kahrs, J., Dichgans, J.: Language processing in aphasia: changes in lateralization patterns during recovery reflect cerebral plasticity in adults. Electroencephalogr. Clin. Neurophysiol. 102(2), 86–97 (1997). doi:
  54. Tupak, S.V., Dresler, T., Badewien, M., Hahn, T., Ernst, L.H., Herrmann, M.J., Fallgatter, A.J.: Inhibitory transcranial magnetic theta burst stimulation attenuates prefrontal cortex oxygenation. Hum. Brain Mapp. 34(1), 150–157 (2013). doi:
  55. Võ, M.L., Conrad, M., Kuchinke, L., Urton, K., Hofmann, M.J., Jacobs, A.M.: The Berlin affective word list reloaded (BAWL-R). Behav. Res. Methods 41(2), 534–538 (2009)CrossRefGoogle Scholar
  56. Watson, D., Clark, L.A., Tellegen, A.: Development and validation of brief measures of positive and negative affect: the PANAS scales. J. Pers. Soc. Psychol. 54(6), 1063–1070 (1988)CrossRefGoogle Scholar
  57. Whittle, S., Yücel, M., Yap, M.B.H., Allen, N.B.: Sex differences in the neural correlates of emotion: evidence from neuroimaging. Biol. Psychol. 87(3), 319–333 (2011). doi:
  58. Wiebking, C., de Greck, M., Duncan, N.W., Heinzel, A., Tempelmann, C., Northoff, G.: Are emotions associated with activity during rest or interoception? An exploratory fMRI study in healthy subjects. Neurosci. Lett. 491(1), 87–92 (2011)CrossRefGoogle Scholar
  59. Williams, J.B.: A structured interview guide for the Hamilton depression rating scale. Arch. Gen. Psychiatry 45(8), 742–747 (1988)CrossRefGoogle Scholar
  60. Yoshida, K., Sawamura, D., Inagaki, Y., Ogawa, K., Ikoma, K., Sakai, S.: Brain activity during the flow experience: a functional near-infrared spectroscopy study. Neurosci. Lett. 573, 30–34 (2014). doi:
  61. Young, R.C., Biggs, J.T., Ziegler, V.E., Meyer, D.A.: A rating scale for mania: reliability, validity and sensitivity. Br. J. Psychiatry 133(5), 429–435 (1978)CrossRefGoogle Scholar
  62. Yuan, Z., Ye, J.: Fusion of fNIRS and fMRI data: identifying when and where hemodynamic signals are changing in human brains. Front. Hum. Neurosci. 7 (2013). doi:

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.M.I.N.D. Lab S.I. Newhouse School of Public CommunicationsSyracuse University School of Information StudiesSyracuseUSA

Personalised recommendations