Deep Learning Techniques in Neuroergonomics

  • Sanghyun Choo
  • Chang S. NamEmail author
Part of the Cognitive Science and Technology book series (CSAT)


There is increasing interest in using deep learning (DL) for neuroergonomics research that investigates the human brain in relation to behavioral performance in natural environments and everyday settings. But a better understanding of how to design and implement DL techniques is still needed for neuroergonomists. Written for novice neuroergonomists as well as experienced investigators, this chapter presents the history of advancements in DL, its concepts, and applications of DL in neuroergonomics research. In addition to artificial neural network (ANN) which is a basic model for DL, this chapter introduces popular DL models such as the multilayer perceptron (MLP), deep belief network (DBN), convolutional neural network (CNN), and recurrent neural networks (RNN). DL-based neuroergonomics research on four main research areas (i.e., mental workload, motor imagery, driving safety, and emotion recognition) will then be reviewed. Insights into how to model and apply DL techniques will be helpful for neuroergonomics researchers, in particular those who are not familiar with DL, but want to predict and classify brain states under various contexts.


  1. Aghajani, H., Garbey, M., & Omurtag, A. (2017). Measuring mental workload with EEG+fNIRS. Frontiers in Human Neuroscience, 11, 1–20.CrossRefGoogle Scholar
  2. Alhagry, S., Aly, A., & El-Khoribi, R. A. (2017). Emotion recognition based on EEG using LSTM recurrent neural network. International Journal of Advanced Computer Science and Applications, 8(10), 355–358.Google Scholar
  3. Aricò, P., Borghini, G., Di Flumeri, G., Colosimo, A., Bonelli, S., Golfetti, A., & Babiloni, F. (2016). Adaptive automation triggered by EEG-based mental workload index: A passive brain-computer interface application in realistic air traffic control environment. Frontiers in Human Neuroscience, 10, 1–13.Google Scholar
  4. Baldwin, C. L., & Penaranda, B. N. (2012). Adaptive training using an artificial neural network and EEG metrics for within- and cross-task workload classification. NeuroImage, 59, 48–56.CrossRefGoogle Scholar
  5. Balkin, T. J., Horrey, W. J., Graeber, R. C., Czeisler, C. A., & Dinges, D. F. (2011). The challenges and opportunities of technological approaches to fatigue management. Accident Analysis and Prevention, 43, 565–572.CrossRefGoogle Scholar
  6. Bashivan, P., & Bidelman, G. M. (2015). Single trial prediction of normal and excessive cognitive load through EEG feature fusion. In Proceedings of IEEE Signal Processing in Medicine and Biology Symposium (pp. 1–5).Google Scholar
  7. Bashivan, P., Rish, I., Yeasin, M., & Codella, N. (2016). Learning representations from EEG with deep recurrent-convolutional neural networks. In International Conference on Learning Representations (ICLR). arXiv:1511.06448.
  8. Borghini, G., Astolfi, L., Vecchiato, G., Mattia, D., & Babiloni, F. (2014). Measuring neurophysiological signals in aircraft pilots and car drivers for the assessment of mental workload, fatigue and drowsiness. Neuroscience and Biobehavioral Reviews, 44, 58–75.CrossRefGoogle Scholar
  9. Chai, R., Ling, S. H., San, P. P., Naik, G. R., Nguyen, T. N., Tran, Y., & Nguyen, H. T. (2017). Improving EEG-based driver fatigue classification using sparse-deep belief networks. Frontiers in Neuroscience, 11.Google Scholar
  10. Chu, Y., Zhao, X., Zou, Y., Xu, W., Han, J., & Zhao, Y. (2018). A decoding scheme for incomplete motor imagery EEG with deep belief network. Frontiers in Neuroscience, 12, 1–17.CrossRefGoogle Scholar
  11. Cinaz, B., Arnrich, B., La Marca, R., & Tröster, G. (2013). Monitoring of mental workload levels during an everyday life office-work scenario. Personal and Ubiquitous Computing, 17, 229–239.CrossRefGoogle Scholar
  12. Daly, J. J., & Huggins, J. E. (2016). Brain-computer interface: Current and emerging rehabilitation applications. Archives of Physical Medicine and Rehabilitation, 96(30), S1–S7.Google Scholar
  13. Deng, L., Hinton, G., & Kingsbury, B. (2013). New types of deep neural network learning for speech recognition and related applications: An overview. In ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing—Proceedings (pp. 8599–8603).Google Scholar
  14. Djemal, R., Bazyed, A. G., Belwafi, K., Gannouni, S., & Kaaniche, W. (2016). Three-class EEG-based motor imagery classification using phase-space reconstruction technique. Brain Sciences, 6(36).Google Scholar
  15. Durantin, G., Scannella, S., Gateau, T., Delorme, A., & Dehais, F. (2016). Processing functional near infrared spectroscopy signal with a Kalman filter to assess working memory during simulated flight. Frontiers in Human Neuroscience, 9, 1–9.CrossRefGoogle Scholar
  16. Friedman, N., Geiger, D., & Goldszmidt, M. (1997). Bayesian network classifier. Machine Learning, 29, 131–163.CrossRefGoogle Scholar
  17. Gao, Y., Lee, H. J., & Mehmood, R. M. (2015). Deep learning of EEG signals for emotion recognition. In 2015 IEEE International Conference on Multimedia and Expo Workshops, ICMEW (pp. 1–5).Google Scholar
  18. Glorot, X., Bordes, A., & Bengio, Y. (2011). Deep sparse rectifier neural networks. In AISTATS (Vol. 15, pp. 315–323).Google Scholar
  19. Graves, A. (2013). Generating sequences with recurrent neural networks (pp. 1–43). arXiv:1308.0850.
  20. Graves, A., & Schmidhuber, J. (2005). Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Networks, 18(5–6), 602–610.CrossRefGoogle Scholar
  21. Guarda, L., López, E., Moura, M., & Ramos, M. (2018). Drowsiness detection using electroencephalography signals : A deep learning based method. In 14th PSAM International Conference on Probabilistic Safety Assessment and Management.Google Scholar
  22. Hattingh, C. J., Ipser, J., Tromp, S. A., Syal, S., Lochner, C., Brooks, S. J., & Stein, D. J. (2013). Functional magnetic resonance imaging during emotion recognition in social anxiety disorder: An activation likelihood meta-analysis. Frontiers in Human Neuroscience, 6, 1–7.Google Scholar
  23. Hefron, R., Borghetti, B., Kabban, C. S., Christensen, J., & Estepp, J. (2018). Cross-participant EEG-based assessment of cognitive workload using multi-path convolutional recurrent neural networks. Sensors, 18(1339).Google Scholar
  24. Hernández, L. G., Mozos, O. M., Ferrández, J. M., & Antelis, J. M. (2018). EEG-based detection of braking intention under different car driving conditions. Frontiers in Neuroinformatics, 12, 1–14.CrossRefGoogle Scholar
  25. Hinton, G. (2002). Training products of experts by minimizing contrastive divergence. Neural Computation, 14(8), 1711–1800.CrossRefGoogle Scholar
  26. Hinton, G. (2010). A practical guide to training restricted boltzmann machines. Momentum, 9(1), 926.Google Scholar
  27. Hinton, G. E., Osindero, S., & Teh, Y. W. (2006). A fast learning algorithm for deep belief nets. Neural Computation, 18(7), 1527–1554.MathSciNetCrossRefGoogle Scholar
  28. Hochreiter, S., & Urgen Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735–1780.CrossRefGoogle Scholar
  29. Horat, S. K., Herrmann, F. R., Favre, G., Terzis, J., Debatisse, D., Merlo, M. C. G., & Missonnier, P. (2016). Assessment of mental workload: A new electrophysiological method based on intra-block averaging of ERP amplitudes. Neuropsychologia, 82, 11–17.Google Scholar
  30. Hung, Y. C., Wang, Y. K., Prasad, M., & Lin, C. T. (2017). Brain dynamic states analysis based on 3D convolutional neural network. In 2017 IEEE International Conference on Systems, Man, and Cybernetics (pp. 222–227).Google Scholar
  31. Johnson, R. R., Popovic, D. P., Olmstead, R. E., Stikic, M., Levendowski, D. J., & Berka, C. (2011). Drowsiness/alertness algorithm development and validation using synchronized EEG and cognitive performance to individualize a generalized model. Biological Psychology, 87, 241–250.CrossRefGoogle Scholar
  32. Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., & Li, F. F. (2014). Large-scale video classification with convolutional neural networks. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (pp. 1725–1732).Google Scholar
  33. Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems (pp. 1–9).Google Scholar
  34. Kuanar, S., Athitsos, V., Pradhan, N., Mishra, A., & Rao, K. R. (2018). Cognitive analysis of working memory load from Eeg, by a deep recurrent neural network. In ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing—Proceedings (pp. 2576–2580).Google Scholar
  35. Kumar, S., Sharma, A., Mamun, K., & Tsunoda, T. (2016). A deep learning approach for motor imagery EEG signal classification. In Proceedings of APWC CSE (pp. 34–39).Google Scholar
  36. Lahane, P., & Sangaiah, A. K. (2015). An approach to eeg based emotion recognition and classification using kernel density estimation. Procedia Computer Science, 48, 574–581.CrossRefGoogle Scholar
  37. Lai, S., Xu, L., Liu, K., & Zhao, J. (2015). Recurrent convolutional neural networks for text classification. In Proceedings of the Conference of the Association for the Advancement of Artificial Intelligence (AAAI).Google Scholar
  38. Lebon, F., Collet, C., & Guillot, A. (2010). Benefits of motor imagery training on muscle strength. The Journal of Strength and Conditioning Research, 24, 1680–1687.CrossRefGoogle Scholar
  39. Lecun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521, 436–444.CrossRefGoogle Scholar
  40. LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-based learning applied to document recognition. In Proceedings of the IEEE (Vol. 86, no. 11, pp. 2278–2324).Google Scholar
  41. Lecun, Y., Henderson, J., Le Cun, Y., Denker, J. S., Henderson, D., Howard, R. E., & Jackel, L. D. (1990). Handwritten digit recognition with a back-propagation network. Advances in Neural Information Processing Systems, 2, 396–404.Google Scholar
  42. Lee, H. K., & Choi, Y. S. (2018). A convolution neural networks scheme for classification of motor imagery EEG based on wavelet time-frequency image. In International Conference on Information Networking (ICOIN) (pp. 906–909).Google Scholar
  43. Lees, M. N., Cosman, J. D., Lee, J. D., Rizzo, M., & Fricke, N. (2010). Translating cognitive neuroscience to the driver’s operational environment: A neuroergonomics approach. American Journal of Psychology, 123(4), 391–411.CrossRefGoogle Scholar
  44. Li, Y., Huang, J., Zhou, H., & Zhong, N. (2017). Human emotion recognition with electroencephalographic multidimensional features by hybrid deep neural networks. Applied Sciences, 7, 1060.CrossRefGoogle Scholar
  45. Li, P., Jiang, W., & Su, F. (2016). Single-channel EEG-based mental fatigue detection based on deep belief network. In Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS, 2016October (pp. 367–370).Google Scholar
  46. Li, Y., Wu, J., & Yang, J. (2011). Developing a logistic regression model with cross-correlation for motor imagery signal recognition. In 2011 IEEE/ICME International Conference on Complex Medical Engineering (pp. 502–507).Google Scholar
  47. Ma, Y., Ding, X., She, Q., Luo, Z., Potter, T., & Zhang, Y. (2016). Classification of motor imagery EEG signals with support vector machines and particle swarm optimization. In Computational and Mathematical Methods in Medicine (pp. 1–8).Google Scholar
  48. McCulloch, W. S., & Pitts, W. H. (1943). A logical calculus of the ideas immanent in nervous activity. Bulletin of Mathematical Biophysics, 5, 115–133.MathSciNetCrossRefGoogle Scholar
  49. Mehta, R. K., & Parasuraman, R. (2013). Neuroergonomics: A review of applications to physical and cognitive work. Frontiers in Human Neuroscience, 7, 1–10.CrossRefGoogle Scholar
  50. Meinel, A., Castaño-Candamil, S., Reis, J., & Tangermann, M. (2016). Pre-trial EEG-based single-trial motor performance prediction to enhance neuroergonomics for a hand force task. Frontiers in Human Neuroscience, 10, 1–17.CrossRefGoogle Scholar
  51. Murugappan, M., Ramachandran, N., & Sazali, Y. (2010). Classification of human emotion from EEG using discrete wavelet transform. Journal of Biomedical Science and Engineering, 3, 390–396.CrossRefGoogle Scholar
  52. Naseer, N., Noori, F. M., Qureshi, N. K., & Hong, K.-S. (2016). Determining optimal feature-combination for LDA classification of functional near-infrared spectroscopy signals in brain-computer interface application. Frontiers in Human Neuroscience, 10, 1–10.CrossRefGoogle Scholar
  53. Ouyang, W., Wang, X., Zeng, X., Qiu, S., Luo, P., Tian, Y., & Tang, X. (2015). DeepID-Net: Deformable deep convolutional neural networks for object detection. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition.Google Scholar
  54. Parasuraman, R. (2003). Neuroergonomics: Research and practice. Theoretical Issues in Ergonomics Science, 4(1–2), 5–20.CrossRefGoogle Scholar
  55. Parasuraman, R., & Rizzo, M. (2007). Neuroergonomics: The brain at work. Oxford; New York: Oxford University Press.Google Scholar
  56. Phan, K. L., Wager, T., Taylor, S. F., & Liberzon, I. (2002). Functional neuroanatomy of emotion: A meta-analysis of emotion activation studies in PET and fMRI. NeuroImage, 16, 331–348.CrossRefGoogle Scholar
  57. Plis, S. M., Hjelm, D. R., Slakhutdinov, R., Allen, E. A., Bockholt, H. J., Long, J. D., … Calhoun, V. D. (2014). Deep learning for neuroimaging: A validation study. Frontiers in Neuroscience, 8, 1–11.Google Scholar
  58. Razzak, M. I., Naz, S., & Zaib, A. (2017). Deep learning for medical image processing: Overview, challenges and the future. arXiv:1704.06825.
  59. Rosipal, R., Peters, B., Göran Kecklund, T. Å., Gruber, G., Woertz, M., Anderer, P., & Dorffner, G. (2007a). EEG-based drivers’ drowsiness monitoring using a hierarchical gaussian mixture model. In Foundations of Augmented Cognition (pp. 294–303).Google Scholar
  60. Rosipal, R., Peters, B., Kecklund, G., Åkerstedt, T., Gruber, G., Woertz, M., & Dorffner, G. (2007b). EEG-based drivers’ drowsiness monitoring using a hierarchical gaussian mixture model. In Proceedings of the HCII2007—Augmented Cognition (pp. 294–303).Google Scholar
  61. Sakhavi, S., & Guan, C. (2017). Convolutional neural network-based transfer learning and knowledge distillation using multi-subject data in motor imagery BCI. In 8th International IEEE EMBS Conference on Neural Engineering (pp. 588–591).Google Scholar
  62. Sharma, N., & Gedeon, T. (2012). Objective measures, sensors and computational techniques for stress recognition and classification: A survey. Computer Methods and Programs in Biomedicine, 108, 1287–1301.CrossRefGoogle Scholar
  63. Smolensky, P. (1986). Information processing in dynamical systems: Foundations of harmony theory. MA, USA: MIT Press Cambridge.Google Scholar
  64. Sohaib, A. T., Qureshi, S., Hagelbäck, J., Hilborn, O., & Jerčić, P. (2013). Evaluating classifiers for emotion recognition using EEG. In Foundations of Augmented Cognition (pp. 492–501). Berlin, Heidelberg: Springer.Google Scholar
  65. Soleymani, M., Asghari-Esfeden, S., Fu, Y., & Pantic, M. (2016). Analysis of EEG signals and facial expressions for continuous emotion detection. IEEE Transactions on Affective Computing, 7(1), 17–28.CrossRefGoogle Scholar
  66. Solhjoo, S., Nasrabadi, A. M., Reza, M., & Golpayegani, H. (2005). Classification of chaotic signals using Hmm classifiers: Eeg-based mental task classification. In Proceedings of European Signal Processing Conference.Google Scholar
  67. Sweller, J., Van Merrienboer, J. J. G., & Paas, F. G. W. C. (1998). Cognitive Architecture and Instructional Design. Educational Psychology Review, 10(3), 251–296.CrossRefGoogle Scholar
  68. Tripathi, S., Acharya, S., Ranti, S., Mittal, S., & Bhattacharya, S. (2017). Using deep and convolutional neural networks for accurate emotion classification on DEAP dataset. In Proceedings of the Twenty-Ninth AAAI Conference on Innovative Applications (pp. 4746–4752).Google Scholar
  69. Uktveris, T., & Jusas, V. (2017). Application of convolutional neural networks to four-class motor imagery classification problem. Information Technology and Control, 46(2), 260–273.CrossRefGoogle Scholar
  70. van Gerven, M., & Bohte, S. (2017). Editorial: Artificial neural networks as models of neural information processing. Frontiers in Computational Neuroscience, 11, 1–2.Google Scholar
  71. Voulodimos, A., Doulamis, N., Bebis, G., & Stathaki, T. (2018). Recent developments in deep learning for engineering applications. In Computational Intelligence and Neuroscience (pp. 1–2).Google Scholar
  72. Wang, Y. K., Jung, T. P., & Lin, C. T. (2015). EEG-based attention tracking during distracted driving. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 23(6), 1085–1094.CrossRefGoogle Scholar
  73. Wang, F., Zhong, S., J. Peng, J. J., & Liu, Y. (2018). Data augmentation for EEG-based emotion recognition with deep convolutional neural networks. In International Conference on Multi-media Modeling (MMM) (pp. 82–93). Springer.Google Scholar
  74. Werbos, P. J. (1974). Beyond regression: New tools for prediction and analysis in the behavioral sciences. Ph.D. thesis. Harvard University.Google Scholar
  75. Werbos, P. J. (1990). Backpropagation through time: What it does and how to do it. Proceedings of IEEE, 78(10), 1550–1560.CrossRefGoogle Scholar
  76. Wu, H., & Gu, X. (2015). Towards dropout training for convolutional neural networks. Neural Networks.Google Scholar
  77. Yang, H., Sakhavi, S., Ang, K. K., & Guan, C. (2015). On the use of convolutional neural networks and augmented CSP features for multi-class motor imagery of EEG signals classification. In Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS (pp. 2620–2623).Google Scholar
  78. Young, T., Hazarika, D., Poria, S., & Cambria, E. (2018). Recent trends in deep learning based natural language processing. IEEE Computational Intelligence Magazine, 13(3), 55–75.Google Scholar
  79. Zeng, H., Yang, C., Kong, G. D., Qin, F., Zhang, J., & Kong, W. (2018). EEG classification of driver mental states by deep learning. Cognitive Neurodynamics, 12(6), 597–606.CrossRefGoogle Scholar
  80. Zhang, J., & Li, S. (2017). A deep learning scheme for mental workload classification based on restricted Boltzmann machines. Cognition, Technology & Work, 19(4), 607–631.CrossRefGoogle Scholar
  81. Zhang, J., Li, S., & Wang, R. (2017a). Pattern recognition of momentary mental workload based on multi-channel electrophysiological data and ensemble convolutional neural networks. Frontiers in Neuroscience, 11, 1–16.Google Scholar
  82. Zhang, J., Yan, C., & Gong, X. (2017b). Deep convolutional neural network for decoding motor imagery based brain computer interface. In 2017 IEEE International Conference on Signal Processing, Communications and Computing, ICSPCC 2017 (pp. 1–5).Google Scholar
  83. Zheng, W. L., & Lu, B. L. (2015). Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks. IEEE Transactions on Autonomous Mental Development, 7(3), 162–175.CrossRefGoogle Scholar
  84. Zhou, J., Meng, M., Gao, Y., Ma, Y., & Zhang, Q. (2018). Classification of motor imagery EEG using wavelet envelope analysis and LSTM networks. In Proceedings of the 30th Chinese Control and Decision Conference, CCDC 2018 (pp. 5600–5605).Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Edward P. Fitts Department of Industrial & Systems EngineeringNorth Carolina State UniversityRaleighUSA

Personalised recommendations