Investigating Brain Dynamics in Industrial Environment – Integrating Mobile EEG and Kinect for Cognitive State Detection of a Worker

  • Pavle Mijović
  • Miloš Milovanović
  • Ivan Gligorijević
  • Vanja Ković
  • Ivana Živanović-Mačužić
  • Bogdan MijovićEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10284)


In the present work we used wearable EEG sensor for recording brain activity during simulated assembly work, in replicated industrial environment. We investigated attention related modalities of P300 ERP component and engagement index (EI), which is extracted from signal power ratios of α, β and θ frequency bands. Simultaneously, we quantified the task unrelated movements, which are previously reported to be related to attention level, in an automated way employing kinectTM sensor. Reaction times were also recorded and investigated. We found that during the monotonous task, both the P300 amplitude and EI decreased as the time of the task progressed. On the other hand, the increase of the task unrelated movement quantity was observed, together with the increase in RTs. These findings lead to conclusion that the monotonous assembly work induces the decrease of attention and engagement of the workers as the task progresses, which is observable in both neural (EEG) and behavioral (RT and unrelated movements) signal modalities. Apart from observing how the attention-related modalities are changing over time, we investigated the functional relationship between the neural and behavioral modalities by using Pearson’s correlation. Since the Person’s correlation coefficients showed the functional relationship between the attention-related modalities, we proposed the creation of the multimodal implicit Human-Computer Interaction (HCI) system, which could acquire and process neural and behavioral data in real-time, with the aim of creating the system that could be aware of the operator’s mental states during the industrial work, consequently improving the operator’s well-being.


Wireless EEG Kinect ERP P300 Attention Neuroergonomics 



This research is financed under EU - FP7 Marie Curie Actions FP7-PEOPLE-2011-ITN.


  1. 1.
    Cerf, V.G.: Cognitive implants. Commun. ACM 57(2), 7 (2014)CrossRefGoogle Scholar
  2. 2.
    Schmidt, A.: Implicit human computer interaction through context. Pers. Technol. 4(2–3), 191–199 (2000)CrossRefGoogle Scholar
  3. 3.
    Tompkins, J.A., et al.: Manufacturing systems (chap. 8). In: Facilities Planning. Wiley (2010)Google Scholar
  4. 4.
    Shappel, S.A., Wiegmann, D.A.: The human factors analysis and classification system–HFACS (No. DOT/FAA/AM-00/7). US Federal Aviation Administration, Office of Aviation Medicine (2000)Google Scholar
  5. 5.
    Bainbridge, L.: Ironies of automation. Automatica 19(6), 775–779 (1983)CrossRefGoogle Scholar
  6. 6.
    Hamrol, A., Kowalik, D., Kujawińsk, A.: Impact of selected work condition factors on quality of manual assembly process. Hum. Factors Ergon. Manuf. Serv. Ind. 21(2), 156–163 (2011)CrossRefGoogle Scholar
  7. 7.
    Obrenovic, Z., Starcevic, D.: Modeling multimodal human-computer interaction. IEEE Comput. 37(9), 65–72 (2004)CrossRefGoogle Scholar
  8. 8.
    Mijović, P., et al.: Towards creation of implicit HCI model for prediction and prevention of operators’ error. In: Proceedings of the 17th International Conference on Human-Computer Interaction, Los Angeles, CA, August 2015Google Scholar
  9. 9.
    Gramann, K., et al.: Cognition in action: imaging brain/body dynamics in mobile humans. Rev. Neurosci. De Gryter 22(6), 593–608 (2011)Google Scholar
  10. 10.
    Stanford, V.: Wearable computing goes live in industry. IEEE Pervasive Comput. 1(4), 14–19 (2002)CrossRefGoogle Scholar
  11. 11.
    Lukowicz, F., et al.: Wearit@ work: toward real-world industrial wearable computing. IEEE Pervasive Comput. 6(4), 8–13 (2007)CrossRefGoogle Scholar
  12. 12.
    Thomas, B.H., Sandor, C.: What wearable augmented reality can do for you. IEEE Pervasive Comput. 8(2), 8–11 (2009)CrossRefGoogle Scholar
  13. 13.
    Wascher, E., Heppner, H., Hoffmann, S.: Towards the measurement of event-related EEG activity in real-life working environments. Int. J. Psychophysiol. 91(1), 3–9 (2014)CrossRefGoogle Scholar
  14. 14.
    Wolpaw, J.P., et al.: Brain-computer interface technology: a review of the first international meeting. IEEE Trans. Rehabil. Eng. 8(2), 164–173 (2000)CrossRefGoogle Scholar
  15. 15.
    Zander, T.O., Kothe, C.: Towards passive brain–computer interfaces: applying brain–computer interface technology to human–machine systems in general. J. Neural Eng. 8(2), 1–5 (2011)Google Scholar
  16. 16.
    Parasuraman, R.: Neuroergonomics: research and practice. Theor. Issues Ergon. Sci. 4(1–2), 5–20 (2003)CrossRefGoogle Scholar
  17. 17.
    Fafrowicz, M., Marek, T.: Quo vadis, neuroergonoics? Ergonomics 50(11), 1941–1949 (2007)CrossRefGoogle Scholar
  18. 18.
    Picton, T.W., et al.: Guidelines for using human event-related potentials to study cognition: recording standards and publication criteria. Psychophysiology 37(02), 127–152 (2000)CrossRefGoogle Scholar
  19. 19.
    Prinzel, L.J., et al.: A closed-loop system for examining psychophysiological measures for adaptive task allocation. Int. J Aviat. psychol. 10(4), 393–410 (2000)CrossRefGoogle Scholar
  20. 20.
    Luck, S.J., Woodman, G.F., Vogel, E.K.: Event-related potential studies of attention. Trends Cogn. sci. 4(11), 432–440 (2000)CrossRefGoogle Scholar
  21. 21.
    Rogé, J., Pebayle, T., Muzet, A.: Variations of the level of vigilance and of behavioral activities during simulated automobile driving. Accid. Anal. Prev. 33(2), 181–186 (2001)CrossRefGoogle Scholar
  22. 22.
    Bigdely-Shamlo, N., Kreutz-Delgado, K., Robbins, K., Miyakoshi, M., Westerfield, M., Bel-Bahar, T., Kothe, C., Hsi, J., Makeig, S.: Hierarchical event descriptor (HED) tags for analysis of event-related EEG studies. In: Global Conference on Signal and Information Processing (GlobalSIP), 2013 IEEE, pp. 1–4. IEEE (2013)Google Scholar
  23. 23.
    Viola, F.C., et al.: Semi-automatic identification of independent components representing EEG artifact. Clin. Neurophysiol. 120(5), 868–877 (2009)CrossRefGoogle Scholar
  24. 24.
    Arnold, V.I.: Mathematical Methods of Classical Mechanics, vol. 60, 2nd edn. Springer Science & Business Media, New York (1989)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Pavle Mijović
    • 1
  • Miloš Milovanović
    • 2
  • Ivan Gligorijević
    • 1
  • Vanja Ković
    • 3
  • Ivana Živanović-Mačužić
    • 4
  • Bogdan Mijović
    • 1
    Email author
  1. 1.mBrainTrain LCCBelgradeSerbia
  2. 2.IT Department, Faculty of Organizational SciencesUniversity of BelgradeBelgradeSerbia
  3. 3.Department for Psychology, Faculty of PhilosophyUniversity of BelgradeBelgradeSerbia
  4. 4.Department of Anatomy and Forensic Medicine Faculty of Medical SciencesUniversity of KragujevacKragujevacSerbia

Personalised recommendations