Establishing Ground Truth on Pyschophysiological Models for Training Machine Learning Algorithms: Options for Ground Truth Proxies

  • Keith BrawnerEmail author
  • Michael W. Boyce
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10284)


One of the core aspects of human-human interaction is the ability to recognize and respond to the emotional and cognitive states of the other person, leaving human-computer interaction systems, at their core, to perform many of the same tasks.


Ground Truth Galvanic Skin Response Electrodermal Activity Facial Action Code System Ground Truth Information 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Alyuz, N., Okur, E., Oktay, E., Genc, U., Aslan, S., Mete, S.E., Stanhill, D., Arnrich, B., Esme, A.A.: Towards an emotional engagement model: can affective states of a learner be automatically detected in a 1:1 learning scenario. In: Proceedings of the 6th Workshop on Personalization Approaches in Learning Environments (PALE 2016), 24th Conference on User Modeling, Adaptation, and Personalization (UMAP 2016), CEUR Workshop Proceedings, this volume (2016)Google Scholar
  2. 2.
    Berka, C., Levendowski, D., Cvetinović, M., Petrović, M., Davis, G., Lumicao, M.P., Živković, V., Olmstead, R.: Real-time analysis of EEG indices of alertness, cognition and memory acquired with a wireless EEG headset. Int. J. Hum.-Comput. Interact. 17(2), 151–170 (2004)CrossRefGoogle Scholar
  3. 3.
    Boyce, M.W., Cruz, D., Sottilare, R.: Interpretative phenomenological analysis for military tactics instruction. In: Kantola, J.I., Barath, T., Nazir, S., Andre, T. (eds.) Advances in Human Factors, Business Management, Training and Education. AISC, vol. 498, pp. 623–634. Springer, Cham (2017). doi: 10.1007/978-3-319-42070-7_58 CrossRefGoogle Scholar
  4. 4.
    Braun, V., Clarke, V.: Using thematic analysis in psychology. Qual. Res. Psychol. 3(2), 77–101 (2006)CrossRefGoogle Scholar
  5. 5.
    Brawner, K.: Data sharing: low-cost sensors for affect and cognition. In: Proceedings of the Educational Data Mining, London, UK (2014)Google Scholar
  6. 6.
    Brawner, K.W.: Modeling learner mood in realtime through biosensors for intelligent tutoring improvements. Department of Electrical Engineering and Computer Science University of Central Florida, p. 500 (2013)Google Scholar
  7. 7.
    Conati, C.: Probabilistic assessment of user’s emotions in educational games. Appl. Artif. Intell. 16(7–8), 555–575 (2002)CrossRefGoogle Scholar
  8. 8.
    Costa, M., Bratt, S.: Truthiness: challenges associated with employing machine learning on neurophysiological sensor data. In: Schmorrow, D.D.D., Fidopiastis, C.M.M. (eds.) AC 2016. LNCS, vol. 9743, pp. 159–164. Springer, Cham (2016). doi: 10.1007/978-3-319-39955-3_15 Google Scholar
  9. 9.
    de Winter, J.C., Happee, R., Martens, M.H., Stanton, N.A.: Effects of adaptive cruise control and highly automated driving on workload and situation awareness: a review of the empirical evidence. Transp. Res. Part F Traffic Psychol. Behav. 27, 196–217 (2014)CrossRefGoogle Scholar
  10. 10.
    Fairclough, S.H., Karran, A.J., Gilleade, K.: Classification accuracy from the perspective of the user: real-time interaction with physiological computing. In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, pp. 3029–3038. ACM (2015)Google Scholar
  11. 11.
    Gupta, R., Khomami Abadi, M., Cárdenes Cabré, J.A., Morreale, F., Falk, T.H., Sebs, N.: A quality adaptive multimodal affect recognition system for user-centric multimedia indexing. In: Proceedings of the 2016 ACM on International Conference on Multimedia Retrieval, pp. 317–320. ACM (2016)Google Scholar
  12. 12.
    Hancock, P.A., Chignell, M.H.: Mental workload dynamics in adaptive interface design. IEEE Trans. Syst. Man Cybern. 18(4), 647–658 (1988)CrossRefGoogle Scholar
  13. 13.
    Hart, S.G., Staveland, L.E.: Development of NASA-TLX (Task Load Index): results of empirical and theoretical research. Adv. Psychol. 52, 139–183 (1988)CrossRefGoogle Scholar
  14. 14.
    Hoskin, R.: The dangers of self-report. In: Science Brainwaves (2012).
  15. 15.
    Jaques, N., Conati, C., Harley, J.M., Azevedo, R.: Predicting affect from gaze data during interaction with an intelligent tutoring system. In: Trausan-Matu, S., Boyer, K.E., Crosby, M., Panourgia, K. (eds.) ITS 2014. LNCS, vol. 8474, pp. 29–38. Springer, Cham (2014). doi: 10.1007/978-3-319-07221-0_4 CrossRefGoogle Scholar
  16. 16.
    Johnson, R.R., Popovic, D.P., Olmstead, R.E., Stikic, M., Levendowski, D.J., Berka, C.: Drowsiness/alertness algorithm development and validation using synchronized EEG and cognitive performance to individualize a generalized model. Biol. Psychol. 87(2), 241–250 (2011)CrossRefGoogle Scholar
  17. 17.
    Kapoor, A., Picard, R.W.: Multimodal affect recognition in learning environments. In: ACM Multimedia 2005, pp. 677–682 (2005)Google Scholar
  18. 18.
    Katsimerou, C., Albeda, J., Huldtgren, A., Heynderickx, I., Redi, J.A.: Crowdsourcing empathetic intelligence: the case of the annotation of EMMA database for emotion and mood recognition. ACM Trans. Intell. Syst. Technol. (TIST) 7(4), 51 (2016)Google Scholar
  19. 19.
    Kennedy, G., Lodge, J.M.: All roads lead to Rome: tracking students’ affect as they overcome misconceptions (2016)Google Scholar
  20. 20.
    Knutson, B., Katovich, K., Suri, G.: Inferring affect from fMRI data. Trends Cogn. Sci. 18(8), 422–428 (2014)CrossRefGoogle Scholar
  21. 21.
    Koelstra, S., Muhl, C., Soleymani, M., Lee, J.-S., Yazdani, A., Ebrahimi, T., Pun, T., Nijholt, A., Patras, I.: Deap: a database for emotion analysis; using physiological signals. IEEE Trans. Affect. Comput. 3(1), 18–31 (2012)CrossRefGoogle Scholar
  22. 22.
    Kokini, C., Carroll, M., Ramirez-Padron, R., Hale, K., Sottilare, R., Goldberg, B.: Quantification of trainee affective and cognitive state in real-time. In: The Interservice/Industry Training, Simulation & Education Conference (I/ITSEC) NTSA, pp. 2155–2166 (2012)Google Scholar
  23. 23.
    McKeown, G., Valstar, M., Cowie, R., Pantic, M., Schroder, M.: The semaine database: annotated multimodal records of emotionally colored conversations between a person and a limited agent. IEEE Trans. Affect. Comput. 3(1), 5–17 (2012)CrossRefGoogle Scholar
  24. 24.
    Muller, M., Guha, S., Baumer, E.P., Mimno, D., Shami, N.S.: Machine learning and grounded theory method: convergence, divergence, and combination. In: Proceedings of the 19th International Conference on Supporting Group Work, pp. 3–8. ACM (2016)Google Scholar
  25. 25.
    Nogueira, P.A., Rodrigues, R., Oliveira, E., Nacke, L.E.: A hybrid approach at emotional state detection: merging theoretical models of emotion with data-driven statistical classifiers. In: Proceedings of the 2013 IEEE/WIC/ACM International Joint Conferences on Web Intelligence (WI) and Intelligent Agent Technologies (IAT)-vol. 02, pp. 253–260. IEEE Computer Society (2013)Google Scholar
  26. 26.
    Pantic, M., Rothkrantz, L.J.: Toward an affect-sensitive multimodal human-computer interaction. Proc. IEEE 91(9), 1370–1390 (2003)CrossRefGoogle Scholar
  27. 27.
    Pedro, M.O., Baker, R., Bowers, A., Heffernan, N.: Predicting college enrollment from student interaction with an intelligent tutoring system in middle school. In: Educational Data Mining 2013 (2013)Google Scholar
  28. 28.
    Picard, R.W., Vyzas, E., Healey, J.: Toward machine emotional intelligence: analysis of affective physiological state. IEEE Trans. Pattern Anal. Mach. Intell. 23(10), 1175–1191 (2001)CrossRefGoogle Scholar
  29. 29.
    Pike, M.F., Maior, H.A., Porcheron, M., Sharples, S.C., Wilson, M.L.: Measuring the effect of think aloud protocols on workload using fNIRS. In: Proceedings of the 32nd Annual ACM Conference on Human Factors in Computing Systems, pp. 3807–3816. ACM (2014)Google Scholar
  30. 30.
    Sim, T., Baker, S., Bsat, M.: The CMU pose, illumination, and expression (PIE) database. In: Proceedings of the Fifth IEEE International Conference on IEEE Automatic Face and Gesture Recognition, pp. 46–51 (2002)Google Scholar
  31. 31.
    Smith, J.A.: Reflecting on the development of interpretative phenomenological analysis and its contribution to qualitative research in psychology. Qual. Res. Psychol. 1(1), 39–54 (2004)Google Scholar
  32. 32.
    Valle, A., Núñez, J.C., Cabanach, R.G., González-Pienda, J.A., Rodríguez, S., Rosário, P., Cerezo, R., Muñoz-Cadavid, M.A.: Self-regulated profiles and academic achievement. Psicothema 20(4), 724–731 (2008)Google Scholar
  33. 33.
    Zhang, L., Jiang, M., Farid, D., Hossain, M.A.: Intelligent facial emotion recognition and semantic-based topic detection for a humanoid robot. Expert Syst. Appl. 40(13), 5160–5168 (2013)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.Army Research LaboratoryOrlandoUSA

Personalised recommendations