Journal of Signal Processing Systems

, Volume 90, Issue 1, pp 99–125 | Cite as

EyeLSD a Robust Approach for Eye Localization and State Detection

  • Benrachou Djamel Eddine
  • Filipe Neves dos Santos
  • Brahim Boulebtateche
  • Salah Bensaoula


Improving the safety of public roads and industrial factories requires more reliable and robust computer vision-based approaches for monitoring the eye state (open or closed) of human operators. Getting this information in real time when humans are driving cars or using hazardous machinery will help to prevent accidents and deaths. This paper proposes a new framework called EyeLSD to localize the eyes and detect their states without face detection step. For EyeLSD aims, two novel descriptors are proposed: enhanced Pyramidal Local Binary Pattern Histogram (ePLBPH) and Multi-Three-Patch LBP histogram (Multi-TPLBP). The performance of EyeLSD with ePLBPH and Multi-TPLBP is evaluated and compared against other approaches. For this evaluation three independent and public datasets were used: BioID, CAS-PEAL-R1 and ZJU datasets. The set EyeLSD, ePLBPH and Multi-TPLBP have a greater performance when compared against the state-of-the-art algorithms. The proposed approach is very stable under large range of eye appearances caused by expression, rotation, lighting, head pose, and occlusion.


Eye localization Eye state measurement Image processing Machine learning 



Project NORTE-01-0145-FEDER-000020” is financed by the North Portugal Regional Operational Programme (NORTE 2020), under the PORTUGAL 2020 Partnership Agreement, and through the European Regional Development Fund (ERDF).


  1. 1.
    (1998). PERCLOS: a valid psychophysiological measure of alertness as assessed by psychomotor vigilance. In Indianapolis, Federal Highway Administration, Office of Motor Carriers. Tech. Rep. MCRT-98-006.Google Scholar
  2. 2.
    Ahonen, T., Hadid, A., & Pietikainen, M. (2006). Face description with local binary patterns: application to face recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(12), 2037–2041.CrossRefMATHGoogle Scholar
  3. 3.
    Benrachou, D. E., Boulebtateche, B., & Bensaoula, S. (2013). Gabor/pca/svm-based face detection for drivers monitoring. Journal of Automation and Control Engineering, 1, 115–118.CrossRefGoogle Scholar
  4. 4.
    Benrachou, D. E., dos Santos, F. N., Boulebtateche, B., & Bensaoula, S. (2015). Automatic eye localization multi-block lbp vs. pyramidal lbp three-levels image decomposition for eye visual appearance description. In Pattern recognition and image analysis (pp. 718–726). Springer.Google Scholar
  5. 5.
    Benrachou, D. E., dos Santos, F. N., Boulebtateche, B., & Bensaoula, S. (2015). Online vision-based eye detection: Lbp/svm vs lbp/lstm-rnm. In CONTROLO’2014–proceedings of the 11th Portuguese conference on automatic control (pp. 659–668). Springer.Google Scholar
  6. 6.
    Chang, C. C., & Lin, C. J. (2011). Libsvm: a library for support vector machines. ACM Transactions on Intelligent Systems and Technology (TIST), 2(3), 27.Google Scholar
  7. 7.
    Cootes, T. F., Taylor, C. J., Cooper, D. H., & Graham, J. (1995). Active shape models-their training and application. Computer Vision and Image Understanding, 61(1), 38–59.CrossRefGoogle Scholar
  8. 8.
    Feng, G. C., & Yuen, P. C. (1998). Variance projection function and its application to eye detection for human face recognition. Pattern Recognition Letters, 19(9), 899–906.CrossRefGoogle Scholar
  9. 9.
    Flores, M. J., Armingol, J. M., & de la Escalera, A. (2010). Real-time warning system for driver drowsiness detection using visual information. Journal of Intelligent & Robotic Systems, 59(2), 103–125.Google Scholar
  10. 10.
    Gao, W., Cao, B., Shan, S., Chen, X., Zhou, D., Zhang, X., & Zhao, D. (2008). The cas-peal large-scale chinese face database and baseline evaluations. IEEE Transactions on Systems, Man and Cybernetics Part a Systems and Humans, 38(1), 149.CrossRefGoogle Scholar
  11. 11.
    Ge, S., Yang, R., Wen, H., Chen, S., & Sun, L. (2014). Eye localization based on correlation filter bank. In 2014 IEEE international conference on multimedia and expo (ICME) (pp. 15). IEEE.Google Scholar
  12. 12.
    Gonzalez, R. C., & Woods, R. E. (2002). Digital image processing, 2nd edn. (pp. 523–532). Englewood Cliffs, NJ: Prentice-Hall.Google Scholar
  13. 13.
    González-Ortega, D., Díaz-Pernas, F., Antón-Rodríguez, M., Martínez-Zarzuela, M., & Díez-Higuera, J. (2013). Real-time vision-based eye state detection for driver alertness monitoring. Pattern Analysis and Applications, 16(3), 285–306.MathSciNetCrossRefGoogle Scholar
  14. 14.
    Harwood, D., Ojala, T., Pietikäinen, M., Kelman, S., & Davis, L. (1995). Texture classification by center-symmetric auto-correlation, using kullback discrimination of distributions. Pattern Recognition Letters, 16(1), 1–10.CrossRefGoogle Scholar
  15. 15.
    Hassaballah, M., Kanazawa, T., & Ido, S. (2010). Efficient eye detection method based on grey intensity variance and independent components analysis. Computer Vision, IET, 4(4), 261–271.CrossRefGoogle Scholar
  16. 16.
    Hassner, T., Harel, S., Paz, E., & Enbar, R. (2015). Effective face frontalization in unconstrained images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 4295–4304).Google Scholar
  17. 17.
    Hollingsworth, K., Clark, S., Thompson, J., Flynn, P. J., & Bowyer, K. W. (2013). Eyebrow segmentation using active shape models, SPIE Defense, Security, and Sensing (pp. 871,208–871,208). International Society for Optics and Photonics.Google Scholar
  18. 18.
    Kalbkhani, H., Shayesteh, M. G., & Mohsen Mousavi, S. (2013). Efficient algorithms for detection of face, eye and eye state. Computer Vision, IET, 7(3), 184–200.CrossRefGoogle Scholar
  19. 19.
    Lazebnik, S., Schmid, C., & Ponce, J. (2006). Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Vol. 2, pp. 2169–2178). IEEE.Google Scholar
  20. 20.
    Lehmann, A., Leibe, B., & Van Gool, L. (2011). Fast prism: Branch and bound hough transform for object class detection. International Journal of Computer Vision, 94(2), 175–197.Google Scholar
  21. 21.
    Leibe, B., Leonardis, A., & Schiele, B. (2008). Robust object detection with interleaved categorization and segmentation. International Journal of Computer Vision, 77(1-3), 259–289.CrossRefGoogle Scholar
  22. 22.
    Li, W., Wang, Y., & Wang, Y. (2011). Eye location via a novel integral projection function and radial symmetry transform. International Journal of Digital Content Technology and its Applications, 5(8), 70–80.CrossRefGoogle Scholar
  23. 23.
    Mäenpää, T. (2003). The Local binary pattern approach to texture analysis: extensions and applications. Oulun Yliopisto.Google Scholar
  24. 24.
    Mäenpää, T., & Pietikäinen, M. (2003). Multi-scale binary patterns for texture analysis. In Image analysis (pp. 885–892). Springer.Google Scholar
  25. 25.
    Mahalingam, G., & Ricanek, Jr K. (2013). Lbp-based periocular recognition on challenging face EURASIP. Journal on Image and Video Processing, 2013(1), 1–13.Google Scholar
  26. 26.
    Ojala, T., Pietikäinen, M., & Harwood, D. (1996). A comparative study of texture measures with classification based on featured distributions. Pattern Recognition, 29(1), 51–59.CrossRefGoogle Scholar
  27. 27.
    Ojala, T., Pietikäinen, M., & Mäenpää, T. (2000). Gray scale and rotation invariant texture classification with local binary patterns. In Computer vision-ECCV 2000 (pp. 404–420). Springer.Google Scholar
  28. 28.
    Ojala, T., Pietikainen, M., & Maenpaa, T. (2002). Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(7), 971–987.CrossRefMATHGoogle Scholar
  29. 29.
    Oyini Mbouna, R., Kong, S. G., & Chun, M. G. (2013). Visual analysis of eye state and head pose for driver alertness monitoring. IEEE Transactions on Intelligent Transportation Systems, 14(3), 1462–1469.CrossRefGoogle Scholar
  30. 30.
    Pang, Y., Yuan, Y., & Li, X. (2008). Gabor-based region covariance matrices for face recognition. IEEE Transactions on Circuits and Systems for Video Technology, 18(7), 989–993.CrossRefGoogle Scholar
  31. 31.
    Qian, X., Hua, X. S., Chen, P., & Ke, L. (2011). Plbp: An effective local binary patterns texture descriptor with pyramid representation. Pattern Recognition, 44(10), 2502–2515.CrossRefGoogle Scholar
  32. 32.
    Pan, G., Sun, L., Wu, Z., & Lao, S. (2007). Eyeblink-based anti-spoofing in face recognition from a generic webcamera. In IEEE 11th international conference on computer vision, 2007. ICCV 2007 (pp. 1–8). IEEE.Google Scholar
  33. 33.
    Ren, Y., Wang, S., Hou, B., & Ma, J. (2014). A novel eye localization method with rotation invariance. IEEE Transactions on Image Processing, 23(1), 226–239.MathSciNetCrossRefMATHGoogle Scholar
  34. 34.
    Shi, Y., Yan, Z., Ge, H., & Mei, L. (2014). Visual objects location based on hand eye coordination. In Future information technology (pp. 403–408). Springer.Google Scholar
  35. 35.
    Sirohey, S., Rosenfeld, A., & Duric, Z. (2002). A method of detecting and tracking irises and eyelids in video. Pattern Recognition, 35(6), 1389–1401.CrossRefMATHGoogle Scholar
  36. 36.
    Smith, B., Brandt, J., Lin, Z., & Zhang, L. (2014). Nonparametric context modeling of local appearance for pose-and expression-robust facial landmark localization. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1741–1748).Google Scholar
  37. 37.
    Song, F., Tan, X., Chen, S., & Zhou, Z. H. (2013). A literature survey on robust and efficient eye localization in real-life scenarios. Pattern Recognition, 46(12), 3157–3173.CrossRefGoogle Scholar
  38. 38.
    Song, F., Tan, X., Liu, X., & Chen, S. (2014). Eyes closeness detection from still images with multi-scale histograms of principal oriented gradients. Pattern Recognition, 47(9), 2825– 2838.CrossRefGoogle Scholar
  39. 39.
    Tan, X., Song, F., Zhou, Z. H., & Chen, S. (2009). Enhanced pictorial structures for precise eye localization under incontrolled conditions. In IEEE conference on computer vision and pattern recognition, 2009. CVPR 2009 (pp. 1621–1628). IEEE.Google Scholar
  40. 40.
    Tan, X., & Triggs, B. (2010). Enhanced local texture feature sets for face recognition under difficult lighting conditions. IEEE Transactions on Image Processing, 19(6), 1635–1650.MathSciNetCrossRefMATHGoogle Scholar
  41. 41.
    Turtinen, M., Pietikäinen, M., Turtinen, M., & Pietikäinen, M. (2006). Contextual analysis of textured scene images. In Proceedings of the 17th British machine vision conference (BMVC 2006) (Vol. 2, pp. 849–858). Edinburgh, UK.Google Scholar
  42. 42.
    Vapnik, V. N., & Vapnik, V. (1998). Statistical learning theory (Vol. 1). New York: Wiley.Google Scholar
  43. 43.
    Viola, P., & Jones, M. (2001). Rapid object detection using a boosted cascade of simple features. In Proceedings of the 2001 IEEE computer society conference on computer vision and pattern recognition, 2001 (Vol. 1, pp. I–511). IEEE.Google Scholar
  44. 44.
    Wang, Q., & Yang, J. (2006). Eye detection in facial images with unconstrained background. Journal of Pattern Recognition Research, 1(1), 55–62.CrossRefGoogle Scholar
  45. 45.
    Wolf, L., Hassner, T., & Taigman, Y. (2008). Descriptor based methods in the wild. In Workshop on faces in ‘real-Life’ images: detection, alignment, and recognition.Google Scholar
  46. 46.
    Wu, J., & Mei, L. (2013). A face recognition algorithm based on asm and gabor features of key points. In 2012 international conference on graphic and image processing (pp. 87,686L–87,686L). International Society for Optics and Photonics.Google Scholar
  47. 47.
    Wu, Y., & Ji, Q. (2014). Learning the deep features for eye detection in uncontrolled conditions. In 22nd international conference on pattern recognition (ICPR) 2014 (pp. 455–459). IEEE.Google Scholar
  48. 48.
    Wu, Y. S., Lee, T. W., Wu, Q. Z., & Liu, H. S. (2010). An eye state recognition method for drowsiness detection. In Vehicular technology conference (VTC 2010-Spring), 2010 IEEE 71st (pp. 1–5). IEEE.Google Scholar
  49. 49.
    Xiang, C., Ding, S. Q., & Lee, T. H. (2005). Geometrical interpretation and architecture selection of mlp. IEEE Transactions on Neural Networks/a Publication of the IEEE Neural Networks Council, 16(1), 84–96.CrossRefGoogle Scholar
  50. 50.
    Xu, C., Zheng, Y., & Wang, Z. (2008). Eye states detection by boosting local binary pattern histogram features. In 15th IEEE international conference on image processing, 2008. ICIP 2008 (pp. 1480–1483). IEEE.Google Scholar
  51. 51.
    Yi, D., Lei, Z., & Li, S. Z. (2011). A robust eye localization method for low quality face images. In International joint conference on biometrics (IJCB), 2011 (pp. 1–6). IEEE.Google Scholar
  52. 52.
    Zhang, B., Shan, S., Chen, X., & Gao, W. (2007). Histogram of gabor phase patterns (hgpp): a novel object representation approach for face recognition. IEEE Transactions on Image Processing, 16(1), 57–68.MathSciNetCrossRefGoogle Scholar
  53. 53.
    Zhao, G., Ahonen, T., Matas, J., & Pietikainen, M. (2012). Rotation-invariant image and video description with local binary pattern features. IEEE Transactions on Image Processing, 21(4), 1465–1477.MathSciNetCrossRefMATHGoogle Scholar
  54. 54.
    Zhou, M., Wang, X., Wang, H., Heo, J., & Nam, D. (2015). Precise eye localization with improved sdm. In IEEE international conference on image processing (ICIP), 2015 (pp. 4466–4470). IEEE.Google Scholar
  55. 55.
    Zhou, Z.H., & Geng, X. (2004). Projection functions for eye detection. Pattern Recognition, 37(5), 1049–1056.CrossRefMATHGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2017

Authors and Affiliations

  • Benrachou Djamel Eddine
    • 1
  • Filipe Neves dos Santos
    • 2
  • Brahim Boulebtateche
    • 1
  • Salah Bensaoula
    • 1
  1. 1.University Badji Mokhtar AnnabaAnnabaAlgeria
  2. 2.INESC TEC FEUP campusPortoPortugal

Personalised recommendations