Mental fatigue level detection based on event related and visual evoked potentials features fusion in virtual indoor environment

  • Hachem A. LamtiEmail author
  • Mohamed Moncef Ben Khelifa
  • Vincent Hugel
Research Article


The purpose of this work is to set up a model that can estimate the mental fatigue of users based on the fusion of relevant features extracted from Positive 300 (P300) and steady state visual evoked potentials (SSVEP) measured by electroencephalogram. To this end, an experimental protocol describes the induction of P300, SSVEP and mental workload (which leads to mental fatigue by varying time-on-task) in different scenarios where environmental artifacts are controlled (obstacles number, obstacles velocities, ambient luminosity). Ten subjects took part in the experiment (with two suffering from cerebral palsy). Their mission is to navigate along a corridor from a starting point A to a goal point B where specific flickering stimuli are introduced to perform the P300 task. On the other hand, SSVEP task is elicited thanks to 10 Hz flickering lights. Correlated features are considered as inputs to fusion block which estimates mental workload. In order to deal with uncertainties and heterogeneity of P300 and SSVEP features, Dempster–Shafer (D–S) evidential reasoning is introduced. As the goal is to assess the reliability for the estimation of mental fatigue levels, D–S is compared to multi layer perception and linear discriminant analysis. The results show that D–S globally outperforms the other classifiers (although its performance significantly decreases between healthy and palsied groups). Finally we discuss the feasibility of such a fusion proposal in real life situation.


BCI Mental fatigue Evidential reasoning P300 SSVEP 


Compliance with ethical standards

Ethical approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.


  1. Banerjee S, Khajanchi S, Chaudhuri S (2015) A mathematical model to elucidate brain tumor abrogation by immunotherapy with T11 target structure. PLoS ONE 40:e123611Google Scholar
  2. Chew LH, Teo J, Mountstephens J (2015) Aesthetic preference recognition of 3D shapes using EEG. Cognit Neurodyn 10:165–173CrossRefGoogle Scholar
  3. Cortese S, Ferrin M, Brandeis D, Buitelaar J, Daley D, Dittmann RW, Holtmann M, Santosh P, Stevenson J, Stringaris A, Zuddas A, Sonuga-Barke EJS (2015) Cognitive training for attention-deficit/hyperactivity disorder: meta-analysis of clinical and neuropsychological outcomes from randomized controlled trials. J Am Acad Child Adolesc Psychiatry 54(3):164–174CrossRefGoogle Scholar
  4. Crippa A, Maurits NM, Lorist MM, Roerdink JBTM (2011) Visual computing in biology and medicine: graph averaging as a means to compare multichannel EEG coherence networks and its application to the study of mental fatigue and neurodegenerative disease. Comput Graph 35(2):265–274. CrossRefGoogle Scholar
  5. Crowley K, Sliney A, Pitt I, Murphy D (2010) Evaluating a brain–computer interface to categorise human emotional response. In: Proceedings of the 2010 10th IEEE International Conference on Advanced Learning Technologies, ICALT ’10. IEEE Computer Society, Washington, pp 276–278Google Scholar
  6. Csatho A, van der Linden D, Darnai G, Hopstaken JF (2013) The same-object benefit is influenced by time-on-task. J Cognit Psychol 25(3):319–327. CrossRefGoogle Scholar
  7. da Silva FL, Niedermeyer E (2004) lectroencphalographie: Principes de base, les applications cliniques, et les domaines connexes. Lippincott Williams and Wilkins. ISBN 0-7817-5126-8, 9780781751261Google Scholar
  8. Dempster AP (2008) A generalization of Bayesian inference. In: Yager RR, Liu L (eds) Classic works of the Dempster–Shafer theory of belief functions, volume 219 of studies in fuzziness and soft computing. Springer, Berlin, pp 73–104Google Scholar
  9. Dey AK, Mann DD (2010) A complete task analysis to measure the workload associated with operating an agricultural sprayer equipped with a navigation device. Appl Ergon 41(1):146–149CrossRefGoogle Scholar
  10. Dornhege G, Blankertz B, Curio G, Müller K-R (2003) Increase information transfer rates in BCI by CSP extension to multi-class. In: NIPSGoogle Scholar
  11. Friman O, Luth T, Volosyak I, Graser A (2007a) Spelling with steady-state visual evoked potentials. In: 3rd International IEEE/EMBS Conference on Neural Engineering, 2007. CNE ’07, pp 354–357Google Scholar
  12. Friman O, Volosyak I, Graser A (2007b) Multiple channel detection of steady-state visual evoked potentials for brain–computer interfaces. IEEE Trans Biomed Eng 54(4):742–750CrossRefGoogle Scholar
  13. Gao X, Xu D, Cheng M, Gao S (2003) A BCI-based environmental controller for the motion-disabled. IEEE Trans Neural Syst Rehabil Eng 11(2):137–140. ISSN 1534-4320.
  14. Ghosh D, Khajanchi S, Mangiarotti S, Denis F, Dana SK, Letellier C (2017) How tumor growth can be influenced by delayed interactions between cancer cells and the microenvironment? Biosystems 158:17–30. ISSN 0303-2647Google Scholar
  15. Girelli M, Luck S (1997) Are the same attentional mechanisms used to detect visual search targets defined by color, orientation, and motion? J Cognit Neurosci 9:238–53CrossRefGoogle Scholar
  16. Grandjean EP, Wotzka G, Schaad R, Gilgen A (1971) Fatigue and stress in air traffic controllers. Ergonomics 14(1):159–165CrossRefGoogle Scholar
  17. Haapalainen E, Kim SJ, Forlizzi JF, Dey AK (2010) Psycho-physiological measures for assessing cognitive load. In: Proceedings of the 12th ACM International Conference on Ubiquitous Computing, Ubicomp ’10. ACM, New York, pp 301–310Google Scholar
  18. Hart SG, Staveland LE (1988) Development of NASA-TLX (task load index): results of empirical and theoretical research. In: Hancock PA, Meshkati N (eds) Human mental workload, volume 52 of advances in psychology. North-Holland, Amsterdam, pp 139–183CrossRefGoogle Scholar
  19. Ho Hwang J, Won Nam K, Jang DP, Kim I (2017) Effects of spectral smearin of stimuli on the performance of auditory steady-state response-based brain–computer interface. Cognit Neurodyn 11:515–527 07CrossRefGoogle Scholar
  20. Huang M, Jin J, Zhang Y, Hu D, Wang X (2017) Usage of drip drops as stimuli in an auditory P300 BCI paradigm. Cognit Neurodyn 12:85–94. CrossRefGoogle Scholar
  21. Jo S, Myung R, Yoon D (2012) Quantitative prediction of mental workload with the ACT-R cognitive architecture. Int J Ind Ergon 42(4):359–370CrossRefGoogle Scholar
  22. Khajanchi S (2017) Uniform persistence and global stability for a brain tumor and immune system interaction. Biophys Rev Lett 12:187–208CrossRefGoogle Scholar
  23. Khajanchi S, Das DK, Kar TK (2018) Dynamics of tuberculosis transmission with exogenous reinfections and endogenous reactivation. Phys A Stat Mech Appl 497:52–71. ISSN 0378-4371Google Scholar
  24. Koelstra S, Muhl C, Soleymani M, Lee J-S, Yazdani A, Ebrahimi T, Pun T, Nijholt A, Patras I (2012) Deap: a database for emotion analysis; using physiological signals. IEEE Trans Affect Comput 3(1):18–31CrossRefGoogle Scholar
  25. Kota S, Miho O, Torres CCC, Sato M, Miki N (2016) Mental fatigue monitoring using a wearable transparent eye detection system. Micromachines 7(2):123Google Scholar
  26. Kthner I, Wriessnegger SC, Mller-Putz GR, Kbler A, Halder S (2014) Effects of mental workload and fatigue on the p300, alpha and theta band power during operation of an erp (p300) brain–computer interface. Biol Psychol 102:118–129CrossRefGoogle Scholar
  27. Lalor EC, Kelly SP, Finucane C, Burke R, Smith R, Reilly RB, McDarby G (2005) Steady-state VEP-based brain–computer interface control in an immersive 3D gaming environment. EURASIP J Adv Signal Process 2005(19):3156–3164CrossRefGoogle Scholar
  28. Lamti HA, Ben Khelifa MM, Alimi AM, Gorce P (2014a) Effect of fatigue on ssvep during virtual wheelchair navigation. J Theor Appl Inf Technol 65:1–10Google Scholar
  29. Lamti HA, Khelifa MMB, Alimi AM, Gorce P (2014b) Influence of mental fatigue on P300 and SSVEP during virtual wheelchair navigation. In: 2014 36th Annual International Conference of the Engineering in Medicine and Biology Society (EMBC). IEEE, pp 1255–1258Google Scholar
  30. Lamti HA, Gorce P, Khelifa MMB, Alimi AM (2016) When mental fatigue maybe characterized by event related potential (p300) during virtual wheelchair navigation. Comput Methods Biomech Biomed Eng 19(16):1749–1759CrossRefGoogle Scholar
  31. Lamti HA, Khelifa MMB, Hugel V (2018) Cerebral and gaze data fusion for wheelchair navigation enhancement: case of distracted users. Robotica.
  32. Lazar NA, Luna B, Sweeney JA, Eddy WF (2002) Combining brains: a survey of methods for statistical pooling of information. NeuroImage 16(2):538–550CrossRefGoogle Scholar
  33. Li X-O, Zhang X-W, Feng H (2003) Real time extraction of visual evoked potentials. In: International conference on neural networks and signal processing, 2003. Proceedings of the 2003, vol 2, pp 1342–1345Google Scholar
  34. Liberati G, Federici S, Pasqualotto E (2015) Extracting neurophysiological signals reflecting users’ emotional and affective responses to bci use: a systematic literature review. NeuroRehabilitation 37:10CrossRefGoogle Scholar
  35. Luck SJ (2005) An introduction to the event-related potential technique (cognitive neuroscience). A Bradford Book, 1 ednGoogle Scholar
  36. Mandel C, Luth T, Laue T, Rofer T, Graser A, Krieg-Bruckner B (2009) Navigating a smart wheelchair with a brain–computer interface interpreting steady-state visual evoked potentials. In: IEEE/RSJ international conference on intelligent robots and systems, 2009. IROS 2009, pp 1118–1125Google Scholar
  37. Meng L-F, Lu C-P, Chen B-W, Chen C-H (2006) Fatigue-induced reversed hemispheric plasticity during motor repetitions: a brain electrophysiological study. In: Proceedings of the 13 international conference on neural information processing—volume part I, ICONIP’06. Springer, Berlin, pp 65–71. ISBN 3-540-46479-4, 978-3-540-46479-2Google Scholar
  38. Molina GG, Tsoneva T, Nijholt A (2009) Emotional brain–computer interfaces. In: 3rd international conference on affective computing and intelligent interaction and workshops, 2009. ACII 2009, pp 87–108Google Scholar
  39. Murata A, Uetake A, Takasawa Y (2005) Evaluation of mental fatigue using feature parameter extracted from event-related potential. Int J Ind Ergon 35(8):761–770CrossRefGoogle Scholar
  40. Myers LJ, Hunter DJ (2009) Parsimonious identification of physiological indices for monitoring cognitive fatigue. In: Proceedings of the 5th international conference on foundations of augmented cognition. Neuroergonomics and Operational Neuroscience: Held As Part of HCI International 2009, FAC ’09. Springer, Berlin, pp 495–503. ISBN 978-3-642-02811-3Google Scholar
  41. O’Donnell RD, Eggemeier FT (1986) Workload assessment methodology. Cognit Process Perform 2:1–49Google Scholar
  42. Perez JLM, Cruz AB (2007) Linear discriminant analysis on brain computer interface. In: IEEE International Symposium on Intelligent Signal Processing, 2007. WISP 2007, pp 1–6Google Scholar
  43. Perlstein WM, Cole MA, Larson M, Kelly K, Seignourel P, Keil A (2003) Steady-state visual evoked potentials reveal frontally-mediated working memory activity in humans. Neurosci Lett 342(3):191–195CrossRefGoogle Scholar
  44. Pimenta A, Carneiro D, Novais P, Neves J (2013) Monitoring mental fatigue through the analysis of keyboard and mouse interaction patterns. In: Pan J-S, Polycarpou MM, Woźniak M, de Carvalho ACPLF, Quintián H, Corchado E (eds) Hybrid artificial intelligent systems. Springer, Berlin, pp 222–231CrossRefGoogle Scholar
  45. Puanhvuan D, Khemmachotikun S, Wechakarn P, Wijarn B, Wongsawat Y (2017) Navigation-synchronized multimodal control wheelchair from brain to alternative assistive technologies for persons with severe disabilities. Cognit Neurodyn 11:117–134CrossRefGoogle Scholar
  46. Queteschiner D (2012) Reality factory.
  47. Regan D (1966) Some characteristics of average steady-state and transient responses evoked by modulated light. Electroencephalogr Clin Neurophysiol 20(3):238–248CrossRefGoogle Scholar
  48. Roy RN, Bonnet S, Charbonnier S, Campagne A (2013) Mental fatigue and working memory load estimation: interaction and implications for EEG-based passive BCI. In: 2013 35th annual international conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp 6607–6610Google Scholar
  49. Ryu K, Myung R (2005) Evaluation of mental workload with a combined measure based on physiological indices during a dual task of tracking and mental arithmetic. Int J Ind Ergon 35(11):991–1009CrossRefGoogle Scholar
  50. Schlesinger M, Amso D, Johnson SP (2007) The neural basis for visual selective attention in young infants: a computational account. Adapt Behav 15(2):135–148CrossRefGoogle Scholar
  51. Valbuena D, Cyriacks M, Friman O, Volosyak I, Graser A (2007) Brain–computer interface for high-level control of rehabilitation robotic systems. In: IEEE 10th international conference on rehabilitation robotics, 2007. ICORR 2007, pp 619–625Google Scholar
  52. Vialatte F-B, Maurice M, Dauwels J, Cichocki A (2010) Steady-state visually evoked potentials: focus on essential paradigms and future perspectives. Prog Neurobiol 90(4):418–438CrossRefGoogle Scholar
  53. Wang Y, Wang R, Gao X, Hong B, Gao S (2006) A practical VEP-based brain–computer interface. IEEE Trans Neural Syst Rehabil Eng 14(2):234–240CrossRefGoogle Scholar
  54. Wu Z, Lai Y, Xia Y, Wu D, Yao D (2008) Stimulator selection in SSVEP-based BCI. Med Eng Phys 30(8):1079–1088. Special Issue (part): Bioengineering in TaiwanGoogle Scholar
  55. Zhang Y, Guo D, Peng X, Zhang Y, Yao D (2016) Robust frequency recognition for ssvep-based bci with temporally local multivariate synchronization index. Cognit Neurodyn 10(6):505–511CrossRefGoogle Scholar
  56. Zhang C, Zheng C, Pei X, Yu X (2007) Estimating mental fatigue based on multichannel linear descriptors analysis. In: Proceedings of the 1st international conference on digital human modeling, ICDHM’07. Springer, Berlin, pp 521–529. ISBN 978-3-540-73318-8.

Copyright information

© Springer Nature B.V. 2019

Authors and Affiliations

  1. 1.COnception de Systemes Mecaniques et Robotiques (COSMER) LaboratoryUniversity of ToulonToulonFrance
  2. 2.Impact de l’Activite Physique sur la Sante (IAPS) LaboratoryUniversity of ToulonToulonFrance

Personalised recommendations