Driver Drowsiness Estimation by Means of Face Depth Map Analysis

  • Paweł ForczmańskiEmail author
  • Kacper Kutelski
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 889)


In the paper a problem of analysing facial images captured by depth sensor is addressed. We focus on evaluating mouth state in order to estimate the drowsiness of the observed person. In order to perform the experiments we collected visual data using standard RGB-D sensor. The imaging environment mimicked the conditions characteristic for driver’s place of work. During the investigations we trained and applied several contemporary general-purpose object detectors known to be accurate when working in visible and thermal spectra, based on Haar-like features, Histogram of Oriented Gradients, and Local Binary Patterns. Having face detected, we apply a heuristic-based approach to evaluate the mouth state and then estimate the drowsiness level. Unlike traditional, visible light-based methods, by using depth map we are able to perform such analysis in the low level of even in the absence of cabin illumination. The experiments performed on video sequences taken in simulated conditions support the final conclusions.


Depth map Face detection Haar–like features Histogram of oriented gradients Local binary patterns Drowsiness evaluation 


  1. 1.
    Alioua, N., Amine, A., Rziza, M.: Driver’s fatigue detection based on yawning extraction. Int. J. Veh. Technol., Article no. 678786 (2014).
  2. 2.
    Azim, T., Jaffar, M.A., Mirza, A.M.: Fully automated real time fatigue detection of drivers through fuzzy expert systems. Appl. Soft Comput. 18, 25–38 (2014)CrossRefGoogle Scholar
  3. 3.
    Burduk, R.: The AdaBoost algorithm with the imprecision determine the weights of the observations. In: Intelligent Information and Database Systems, Part II, LNCS, vol. 8398, pp. 110–116 (2014)Google Scholar
  4. 4.
    Chang, H., Koschan, A., Abidi, M., Kong, S.G., Won, C.-H.: Multispectral visible and infrared imaging for face recognition. In: 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, pp. 1–6 (2008)Google Scholar
  5. 5.
    Craye, C., Rashwan, A., Kamel, M.S., Karray, F.: A multi-modal driver fatigue and distraction assessment system. Int. J. Intel. Transp. Syst. Res. 14(3), 173–194 (2016)Google Scholar
  6. 6.
    Cyganek, B., Gruszczynski, S.: Hybrid computer vision system for drivers’ eye recognition and fatigue monitoring. Neurocomputing 126, 78–94 (2014)CrossRefGoogle Scholar
  7. 7.
    Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 886–893 (2005)Google Scholar
  8. 8.
    Forczmański, P., Kukharev, G.: Comparative analysis of simple facial features extractors. J. R. Time Image Process. 1(4), 239–255 (2007)CrossRefGoogle Scholar
  9. 9.
    Forczmański, P., Kukharev, G., Shchegoleva, N.: Simple and robust facial portraits recognition under variable lighting conditions based on two-dimensional orthogonal transformations. In: 7th International Conference on Image Analysis and Processing (ICIAP). LNCS, vol. 8156, pp. 602–611 (2013)Google Scholar
  10. 10.
    Forczmański, P.: Human face detection in thermal images using an ensemble of cascading classifiers. In: Hard and Soft Computing for Artificial Intelligence, Multimedia and Security, Advances in Intelligent Systems and Computing, vol. 534, pp. 205–215 (2016)Google Scholar
  11. 11.
    Forczmański, P.: Performance evaluation of selected thermal imaging-based human face detectors. In: Proceedings of the 10th International Conference on Computer Recognition Systems CORES 2017. Advances in Intelligent Systems and Computing, vol. 578, pp. 170–181 (2018)Google Scholar
  12. 12.
    Fornalczyk, K., Wojciechowski, A.: Robust face model based approach to head pose estimation. In: Proceedings of the 2017 Federated Conference on Computer Science and Information Systems, FedCSIS 2017, pp. 1291–1295 (2017)Google Scholar
  13. 13.
    Freund, Y., Schapire, R.E.: A decision-theoretic generalization of on-line learning and an application to boosting. In: Proceedings of the 2nd European Conference on Computational Learning Theory, pp. 23–37 (1995)Google Scholar
  14. 14.
    Fu, R., Wang, H., Zhao, W.: Dynamic driver fatigue detection using hidden Markov model in real driving condition. Exp. Syst. Appl. 63, 397–411 (2016)CrossRefGoogle Scholar
  15. 15.
    Intel RealSense Camera SR300 – Embedded Coded Light 3D Imaging System with Full High Definition Color Camera Product Datasheet, rev. 1 (2016). Accessed 05 Oct 2018
  16. 16.
    Jo, J., Lee, S.J., Park, K.R., Kim, I.J., Kim, J.: Detecting driver drowsiness using feature-level fusion and user-specific classification. Exp. Syst. Appl. 41(4), 1139–1152 (2014)CrossRefGoogle Scholar
  17. 17.
    Kong, W., Zhou, L., Wang, Y., Zhang, J., Liu, J., Gao, S.: A system of driving fatigue detection based on machine vision and its application on smart device. J. Sens. 2015, 11 pages (2015)Google Scholar
  18. 18.
    Krishnasree, V., Balaji, N., Rao, P.S.: A real time improved driver fatigue monitoring system. WSEAS Trans. Signal Process. 10, 146–155 (2014)Google Scholar
  19. 19.
    Nowosielski, A.: Vision-based solutions for driver assistance. J. Theor. Appl. Comput. Sci. 8(4), 35–44 (2014)Google Scholar
  20. 20.
    Makowiec-Dabrowska, T., Siedlecka, J., Gadzicka, E., Szyjkowska, A., Dania, M., Viebig, P., Kosobudzki, M., Bortkiewicz, A.: The work fatigue for drivers of city buses. Medycyna Pracy 66(5), 661–677 (2015)CrossRefGoogle Scholar
  21. 21.
    Małecki, K., Nowosielski, A., Forczmański, P.: Multispectral data acquisition in the assessment of driver’s fatigue. In: Mikulski, J. (ed.) Smart Solutions in Today’s Transport, TST 2017. Communications in Computer and Information Science, vol. 715. pp. 320–332 (2017)Google Scholar
  22. 22.
    Mitas, A., Czapla, Z., Bugdol, M., Ryguła, A.: Registration and evaluation of biometric parameters of the driver to improve road safety, pp. 71–79. Scientific Papers of Transport, Silesian University of Technology (2010)Google Scholar
  23. 23.
    Ojala, T., Pietikinen, M., Harwood, D.: Performance evaluation of texture measures with classification based on Kullback discrimination of distributions. In: Proceedings of the 12th International Conference on Pattern Recognition, vol. 1, pp. 582–585 (1994)Google Scholar
  24. 24.
    Smiatacz, M.: Liveness measurements using optical flow for biometric person authentication. Metrol. Meas. Syst. 19(2), 257–268 (2012)CrossRefGoogle Scholar
  25. 25.
    Staniucha, R., Wojciechowski, A.: Mouth features extraction for emotion classification. In: Proceedings of the 2016 Federated Conference on Computer Science and Information Systems, FedCSIS 2016, pp. 1685–1692 (2016)Google Scholar
  26. 26.
    Viola, P., Jones, M.J.: Robust real-time face detection. Int. J. Comput. Vis. 57(2), 137–154 (2004)CrossRefGoogle Scholar
  27. 27.
    Zhang, Y., Hua, C.: Driver fatigue recognition based on facial expression analysis using local binary patterns. Opt. Int. J. Light. Electron Opt. 126(23), 4501–4505 (2015)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Faculty of Computer Science and Information TechnologyWest Pomeranian University of Technology, SzczecinSzczecinPoland

Personalised recommendations