Skip to main content
SpringerLink
Log in

Deep Learning Techniques in Neuroergonomics

  • Chapter
  • First Online:
Neuroergonomics

Part of the book series: Cognitive Science and Technology ((CSAT))

  • 926 Accesses

Abstract

There is increasing interest in using deep learning (DL) for neuroergonomics research that investigates the human brain in relation to behavioral performance in natural environments and everyday settings. But a better understanding of how to design and implement DL techniques is still needed for neuroergonomists. Written for novice neuroergonomists as well as experienced investigators, this chapter presents the history of advancements in DL, its concepts, and applications of DL in neuroergonomics research. In addition to artificial neural network (ANN) which is a basic model for DL, this chapter introduces popular DL models such as the multilayer perceptron (MLP), deep belief network (DBN), convolutional neural network (CNN), and recurrent neural networks (RNN). DL-based neuroergonomics research on four main research areas (i.e., mental workload, motor imagery, driving safety, and emotion recognition) will then be reviewed. Insights into how to model and apply DL techniques will be helpful for neuroergonomics researchers, in particular those who are not familiar with DL, but want to predict and classify brain states under various contexts.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  • Aghajani, H., Garbey, M., & Omurtag, A. (2017). Measuring mental workload with EEG+fNIRS. Frontiers in Human Neuroscience, 11, 1–20.

    Article  Google Scholar 

  • Alhagry, S., Aly, A., & El-Khoribi, R. A. (2017). Emotion recognition based on EEG using LSTM recurrent neural network. International Journal of Advanced Computer Science and Applications, 8(10), 355–358.

    Google Scholar 

  • Aricò, P., Borghini, G., Di Flumeri, G., Colosimo, A., Bonelli, S., Golfetti, A., & Babiloni, F. (2016). Adaptive automation triggered by EEG-based mental workload index: A passive brain-computer interface application in realistic air traffic control environment. Frontiers in Human Neuroscience, 10, 1–13.

    Google Scholar 

  • Baldwin, C. L., & Penaranda, B. N. (2012). Adaptive training using an artificial neural network and EEG metrics for within- and cross-task workload classification. NeuroImage, 59, 48–56.

    Article  Google Scholar 

  • Balkin, T. J., Horrey, W. J., Graeber, R. C., Czeisler, C. A., & Dinges, D. F. (2011). The challenges and opportunities of technological approaches to fatigue management. Accident Analysis and Prevention, 43, 565–572.

    Article  Google Scholar 

  • Bashivan, P., & Bidelman, G. M. (2015). Single trial prediction of normal and excessive cognitive load through EEG feature fusion. In Proceedings of IEEE Signal Processing in Medicine and Biology Symposium (pp. 1–5).

    Google Scholar 

  • Bashivan, P., Rish, I., Yeasin, M., & Codella, N. (2016). Learning representations from EEG with deep recurrent-convolutional neural networks. In International Conference on Learning Representations (ICLR). arXiv:1511.06448.

  • Borghini, G., Astolfi, L., Vecchiato, G., Mattia, D., & Babiloni, F. (2014). Measuring neurophysiological signals in aircraft pilots and car drivers for the assessment of mental workload, fatigue and drowsiness. Neuroscience and Biobehavioral Reviews, 44, 58–75.

    Article  Google Scholar 

  • Chai, R., Ling, S. H., San, P. P., Naik, G. R., Nguyen, T. N., Tran, Y., & Nguyen, H. T. (2017). Improving EEG-based driver fatigue classification using sparse-deep belief networks. Frontiers in Neuroscience, 11.

    Google Scholar 

  • Chu, Y., Zhao, X., Zou, Y., Xu, W., Han, J., & Zhao, Y. (2018). A decoding scheme for incomplete motor imagery EEG with deep belief network. Frontiers in Neuroscience, 12, 1–17.

    Article  Google Scholar 

  • Cinaz, B., Arnrich, B., La Marca, R., & Tröster, G. (2013). Monitoring of mental workload levels during an everyday life office-work scenario. Personal and Ubiquitous Computing, 17, 229–239.

    Article  Google Scholar 

  • Daly, J. J., & Huggins, J. E. (2016). Brain-computer interface: Current and emerging rehabilitation applications. Archives of Physical Medicine and Rehabilitation, 96(30), S1–S7.

    Google Scholar 

  • Deng, L., Hinton, G., & Kingsbury, B. (2013). New types of deep neural network learning for speech recognition and related applications: An overview. In ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing—Proceedings (pp. 8599–8603).

    Google Scholar 

  • Djemal, R., Bazyed, A. G., Belwafi, K., Gannouni, S., & Kaaniche, W. (2016). Three-class EEG-based motor imagery classification using phase-space reconstruction technique. Brain Sciences, 6(36).

    Google Scholar 

  • Durantin, G., Scannella, S., Gateau, T., Delorme, A., & Dehais, F. (2016). Processing functional near infrared spectroscopy signal with a Kalman filter to assess working memory during simulated flight. Frontiers in Human Neuroscience, 9, 1–9.

    Article  Google Scholar 

  • Friedman, N., Geiger, D., & Goldszmidt, M. (1997). Bayesian network classifier. Machine Learning, 29, 131–163.

    Article  Google Scholar 

  • Gao, Y., Lee, H. J., & Mehmood, R. M. (2015). Deep learning of EEG signals for emotion recognition. In 2015 IEEE International Conference on Multimedia and Expo Workshops, ICMEW (pp. 1–5).

    Google Scholar 

  • Glorot, X., Bordes, A., & Bengio, Y. (2011). Deep sparse rectifier neural networks. In AISTATS (Vol. 15, pp. 315–323).

    Google Scholar 

  • Graves, A. (2013). Generating sequences with recurrent neural networks (pp. 1–43). arXiv:1308.0850.

  • Graves, A., & Schmidhuber, J. (2005). Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Networks, 18(5–6), 602–610.

    Article  Google Scholar 

  • Guarda, L., López, E., Moura, M., & Ramos, M. (2018). Drowsiness detection using electroencephalography signals : A deep learning based method. In 14th PSAM International Conference on Probabilistic Safety Assessment and Management.

    Google Scholar 

  • Hattingh, C. J., Ipser, J., Tromp, S. A., Syal, S., Lochner, C., Brooks, S. J., & Stein, D. J. (2013). Functional magnetic resonance imaging during emotion recognition in social anxiety disorder: An activation likelihood meta-analysis. Frontiers in Human Neuroscience, 6, 1–7.

    Google Scholar 

  • Hefron, R., Borghetti, B., Kabban, C. S., Christensen, J., & Estepp, J. (2018). Cross-participant EEG-based assessment of cognitive workload using multi-path convolutional recurrent neural networks. Sensors, 18(1339).

    Google Scholar 

  • Hernández, L. G., Mozos, O. M., Ferrández, J. M., & Antelis, J. M. (2018). EEG-based detection of braking intention under different car driving conditions. Frontiers in Neuroinformatics, 12, 1–14.

    Article  Google Scholar 

  • Hinton, G. (2002). Training products of experts by minimizing contrastive divergence. Neural Computation, 14(8), 1711–1800.

    Article  Google Scholar 

  • Hinton, G. (2010). A practical guide to training restricted boltzmann machines. Momentum, 9(1), 926.

    Google Scholar 

  • Hinton, G. E., Osindero, S., & Teh, Y. W. (2006). A fast learning algorithm for deep belief nets. Neural Computation, 18(7), 1527–1554.

    Article  MathSciNet  Google Scholar 

  • Hochreiter, S., & Urgen Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735–1780.

    Article  Google Scholar 

  • Horat, S. K., Herrmann, F. R., Favre, G., Terzis, J., Debatisse, D., Merlo, M. C. G., & Missonnier, P. (2016). Assessment of mental workload: A new electrophysiological method based on intra-block averaging of ERP amplitudes. Neuropsychologia, 82, 11–17.

    Google Scholar 

  • Hung, Y. C., Wang, Y. K., Prasad, M., & Lin, C. T. (2017). Brain dynamic states analysis based on 3D convolutional neural network. In 2017 IEEE International Conference on Systems, Man, and Cybernetics (pp. 222–227).

    Google Scholar 

  • Johnson, R. R., Popovic, D. P., Olmstead, R. E., Stikic, M., Levendowski, D. J., & Berka, C. (2011). Drowsiness/alertness algorithm development and validation using synchronized EEG and cognitive performance to individualize a generalized model. Biological Psychology, 87, 241–250.

    Article  Google Scholar 

  • Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., & Li, F. F. (2014). Large-scale video classification with convolutional neural networks. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (pp. 1725–1732).

    Google Scholar 

  • Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems (pp. 1–9).

    Google Scholar 

  • Kuanar, S., Athitsos, V., Pradhan, N., Mishra, A., & Rao, K. R. (2018). Cognitive analysis of working memory load from Eeg, by a deep recurrent neural network. In ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing—Proceedings (pp. 2576–2580).

    Google Scholar 

  • Kumar, S., Sharma, A., Mamun, K., & Tsunoda, T. (2016). A deep learning approach for motor imagery EEG signal classification. In Proceedings of APWC CSE (pp. 34–39).

    Google Scholar 

  • Lahane, P., & Sangaiah, A. K. (2015). An approach to eeg based emotion recognition and classification using kernel density estimation. Procedia Computer Science, 48, 574–581.

    Article  Google Scholar 

  • Lai, S., Xu, L., Liu, K., & Zhao, J. (2015). Recurrent convolutional neural networks for text classification. In Proceedings of the Conference of the Association for the Advancement of Artificial Intelligence (AAAI).

    Google Scholar 

  • Lebon, F., Collet, C., & Guillot, A. (2010). Benefits of motor imagery training on muscle strength. The Journal of Strength and Conditioning Research, 24, 1680–1687.

    Article  Google Scholar 

  • Lecun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521, 436–444.

    Article  Google Scholar 

  • LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-based learning applied to document recognition. In Proceedings of the IEEE (Vol. 86, no. 11, pp. 2278–2324).

    Google Scholar 

  • Lecun, Y., Henderson, J., Le Cun, Y., Denker, J. S., Henderson, D., Howard, R. E., & Jackel, L. D. (1990). Handwritten digit recognition with a back-propagation network. Advances in Neural Information Processing Systems, 2, 396–404.

    Google Scholar 

  • Lee, H. K., & Choi, Y. S. (2018). A convolution neural networks scheme for classification of motor imagery EEG based on wavelet time-frequency image. In International Conference on Information Networking (ICOIN) (pp. 906–909).

    Google Scholar 

  • Lees, M. N., Cosman, J. D., Lee, J. D., Rizzo, M., & Fricke, N. (2010). Translating cognitive neuroscience to the driver’s operational environment: A neuroergonomics approach. American Journal of Psychology, 123(4), 391–411.

    Article  Google Scholar 

  • Li, Y., Huang, J., Zhou, H., & Zhong, N. (2017). Human emotion recognition with electroencephalographic multidimensional features by hybrid deep neural networks. Applied Sciences, 7, 1060.

    Article  Google Scholar 

  • Li, P., Jiang, W., & Su, F. (2016). Single-channel EEG-based mental fatigue detection based on deep belief network. In Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS, 2016–October (pp. 367–370).

    Google Scholar 

  • Li, Y., Wu, J., & Yang, J. (2011). Developing a logistic regression model with cross-correlation for motor imagery signal recognition. In 2011 IEEE/ICME International Conference on Complex Medical Engineering (pp. 502–507).

    Google Scholar 

  • Ma, Y., Ding, X., She, Q., Luo, Z., Potter, T., & Zhang, Y. (2016). Classification of motor imagery EEG signals with support vector machines and particle swarm optimization. In Computational and Mathematical Methods in Medicine (pp. 1–8).

    Google Scholar 

  • McCulloch, W. S., & Pitts, W. H. (1943). A logical calculus of the ideas immanent in nervous activity. Bulletin of Mathematical Biophysics, 5, 115–133.

    Article  MathSciNet  Google Scholar 

  • Mehta, R. K., & Parasuraman, R. (2013). Neuroergonomics: A review of applications to physical and cognitive work. Frontiers in Human Neuroscience, 7, 1–10.

    Article  Google Scholar 

  • Meinel, A., Castaño-Candamil, S., Reis, J., & Tangermann, M. (2016). Pre-trial EEG-based single-trial motor performance prediction to enhance neuroergonomics for a hand force task. Frontiers in Human Neuroscience, 10, 1–17.

    Article  Google Scholar 

  • Murugappan, M., Ramachandran, N., & Sazali, Y. (2010). Classification of human emotion from EEG using discrete wavelet transform. Journal of Biomedical Science and Engineering, 3, 390–396.

    Article  Google Scholar 

  • Naseer, N., Noori, F. M., Qureshi, N. K., & Hong, K.-S. (2016). Determining optimal feature-combination for LDA classification of functional near-infrared spectroscopy signals in brain-computer interface application. Frontiers in Human Neuroscience, 10, 1–10.

    Article  Google Scholar 

  • Ouyang, W., Wang, X., Zeng, X., Qiu, S., Luo, P., Tian, Y., & Tang, X. (2015). DeepID-Net: Deformable deep convolutional neural networks for object detection. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

    Google Scholar 

  • Parasuraman, R. (2003). Neuroergonomics: Research and practice. Theoretical Issues in Ergonomics Science, 4(1–2), 5–20.

    Article  Google Scholar 

  • Parasuraman, R., & Rizzo, M. (2007). Neuroergonomics: The brain at work. Oxford; New York: Oxford University Press.

    Google Scholar 

  • Phan, K. L., Wager, T., Taylor, S. F., & Liberzon, I. (2002). Functional neuroanatomy of emotion: A meta-analysis of emotion activation studies in PET and fMRI. NeuroImage, 16, 331–348.

    Article  Google Scholar 

  • Plis, S. M., Hjelm, D. R., Slakhutdinov, R., Allen, E. A., Bockholt, H. J., Long, J. D., … Calhoun, V. D. (2014). Deep learning for neuroimaging: A validation study. Frontiers in Neuroscience, 8, 1–11.

    Google Scholar 

  • Razzak, M. I., Naz, S., & Zaib, A. (2017). Deep learning for medical image processing: Overview, challenges and the future. arXiv:1704.06825.

  • Rosipal, R., Peters, B., Göran Kecklund, T. Å., Gruber, G., Woertz, M., Anderer, P., & Dorffner, G. (2007a). EEG-based drivers’ drowsiness monitoring using a hierarchical gaussian mixture model. In Foundations of Augmented Cognition (pp. 294–303).

    Google Scholar 

  • Rosipal, R., Peters, B., Kecklund, G., Åkerstedt, T., Gruber, G., Woertz, M., & Dorffner, G. (2007b). EEG-based drivers’ drowsiness monitoring using a hierarchical gaussian mixture model. In Proceedings of the HCII2007—Augmented Cognition (pp. 294–303).

    Google Scholar 

  • Sakhavi, S., & Guan, C. (2017). Convolutional neural network-based transfer learning and knowledge distillation using multi-subject data in motor imagery BCI. In 8th International IEEE EMBS Conference on Neural Engineering (pp. 588–591).

    Google Scholar 

  • Sharma, N., & Gedeon, T. (2012). Objective measures, sensors and computational techniques for stress recognition and classification: A survey. Computer Methods and Programs in Biomedicine, 108, 1287–1301.

    Article  Google Scholar 

  • Smolensky, P. (1986). Information processing in dynamical systems: Foundations of harmony theory. MA, USA: MIT Press Cambridge.

    Google Scholar 

  • Sohaib, A. T., Qureshi, S., Hagelbäck, J., Hilborn, O., & Jerčić, P. (2013). Evaluating classifiers for emotion recognition using EEG. In Foundations of Augmented Cognition (pp. 492–501). Berlin, Heidelberg: Springer.

    Google Scholar 

  • Soleymani, M., Asghari-Esfeden, S., Fu, Y., & Pantic, M. (2016). Analysis of EEG signals and facial expressions for continuous emotion detection. IEEE Transactions on Affective Computing, 7(1), 17–28.

    Article  Google Scholar 

  • Solhjoo, S., Nasrabadi, A. M., Reza, M., & Golpayegani, H. (2005). Classification of chaotic signals using Hmm classifiers: Eeg-based mental task classification. In Proceedings of European Signal Processing Conference.

    Google Scholar 

  • Sweller, J., Van Merrienboer, J. J. G., & Paas, F. G. W. C. (1998). Cognitive Architecture and Instructional Design. Educational Psychology Review, 10(3), 251–296.

    Article  Google Scholar 

  • Tripathi, S., Acharya, S., Ranti, S., Mittal, S., & Bhattacharya, S. (2017). Using deep and convolutional neural networks for accurate emotion classification on DEAP dataset. In Proceedings of the Twenty-Ninth AAAI Conference on Innovative Applications (pp. 4746–4752).

    Google Scholar 

  • Uktveris, T., & Jusas, V. (2017). Application of convolutional neural networks to four-class motor imagery classification problem. Information Technology and Control, 46(2), 260–273.

    Article  Google Scholar 

  • van Gerven, M., & Bohte, S. (2017). Editorial: Artificial neural networks as models of neural information processing. Frontiers in Computational Neuroscience, 11, 1–2.

    Google Scholar 

  • Voulodimos, A., Doulamis, N., Bebis, G., & Stathaki, T. (2018). Recent developments in deep learning for engineering applications. In Computational Intelligence and Neuroscience (pp. 1–2).

    Google Scholar 

  • Wang, Y. K., Jung, T. P., & Lin, C. T. (2015). EEG-based attention tracking during distracted driving. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 23(6), 1085–1094.

    Article  Google Scholar 

  • Wang, F., Zhong, S., J. Peng, J. J., & Liu, Y. (2018). Data augmentation for EEG-based emotion recognition with deep convolutional neural networks. In International Conference on Multi-media Modeling (MMM) (pp. 82–93). Springer.

    Google Scholar 

  • Werbos, P. J. (1974). Beyond regression: New tools for prediction and analysis in the behavioral sciences. Ph.D. thesis. Harvard University.

    Google Scholar 

  • Werbos, P. J. (1990). Backpropagation through time: What it does and how to do it. Proceedings of IEEE, 78(10), 1550–1560.

    Article  Google Scholar 

  • Wu, H., & Gu, X. (2015). Towards dropout training for convolutional neural networks. Neural Networks.

    Google Scholar 

  • Yang, H., Sakhavi, S., Ang, K. K., & Guan, C. (2015). On the use of convolutional neural networks and augmented CSP features for multi-class motor imagery of EEG signals classification. In Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS (pp. 2620–2623).

    Google Scholar 

  • Young, T., Hazarika, D., Poria, S., & Cambria, E. (2018). Recent trends in deep learning based natural language processing. IEEE Computational Intelligence Magazine, 13(3), 55–75.

    Google Scholar 

  • Zeng, H., Yang, C., Kong, G. D., Qin, F., Zhang, J., & Kong, W. (2018). EEG classification of driver mental states by deep learning. Cognitive Neurodynamics, 12(6), 597–606.

    Article  Google Scholar 

  • Zhang, J., & Li, S. (2017). A deep learning scheme for mental workload classification based on restricted Boltzmann machines. Cognition, Technology & Work, 19(4), 607–631.

    Article  Google Scholar 

  • Zhang, J., Li, S., & Wang, R. (2017a). Pattern recognition of momentary mental workload based on multi-channel electrophysiological data and ensemble convolutional neural networks. Frontiers in Neuroscience, 11, 1–16.

    Google Scholar 

  • Zhang, J., Yan, C., & Gong, X. (2017b). Deep convolutional neural network for decoding motor imagery based brain computer interface. In 2017 IEEE International Conference on Signal Processing, Communications and Computing, ICSPCC 2017 (pp. 1–5).

    Google Scholar 

  • Zheng, W. L., & Lu, B. L. (2015). Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks. IEEE Transactions on Autonomous Mental Development, 7(3), 162–175.

    Article  Google Scholar 

  • Zhou, J., Meng, M., Gao, Y., Ma, Y., & Zhang, Q. (2018). Classification of motor imagery EEG using wavelet envelope analysis and LSTM networks. In Proceedings of the 30th Chinese Control and Decision Conference, CCDC 2018 (pp. 5600–5605).

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Edward P. Fitts Department of Industrial & Systems Engineering, North Carolina State University, Raleigh, NC, 27695, USA

    Sanghyun Choo & Chang S. Nam

Authors
  1. Sanghyun Choo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chang S. Nam
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chang S. Nam .

Editor information

Editors and Affiliations

  1. Department of Industrial and Systems Engineering, North Carolina State University, Raleigh, NC, USA

    Chang S. Nam

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Choo, S., Nam, C.S. (2020). Deep Learning Techniques in Neuroergonomics. In: Nam, C. (eds) Neuroergonomics. Cognitive Science and Technology. Springer, Cham. https://doi.org/10.1007/978-3-030-34784-0_7

Download citation

Publish with us

Policies and ethics

Search

Navigation