Skip to main content
SpringerLink
Log in

Real-time eye blink and wink detection for object selection in HCI systems

  • Original Paper
  • Published:
Journal on Multimodal User Interfaces Aims and scope Submit manuscript

Abstract

This paper presents an approach for real-time detection of three types of eye blinks: eye blink (blinking both eyes simultaneously), left and right winks. The process of blink detection has been divided into four parts viz. face localization in facial images acquired through a video camera, eye pair localization, pixels’ motion analysis using optical flow technique, and classification of eye blinks. Blink detection has been performed using a video camera and MATLAB software with image processing and computer vision toolbox. The algorithm takes about 60 ms time for processing a frame and 250 ms time for confirmation and classification of the detected blink. An experiment was conducted to evaluate the performance of the proposed approach in which 10 users voluntarily participated. The performance of the proposed method has been tested under two lighting conditions: natural lighting conditions and controlled lighting conditions. Also, the performance has been tested by varying the distance of the user from the camera. Here, it is observed that the system gives best performance when used under controlled lighting conditions and the user sitting at a distance of about 0.5 m. Accuracy of the proposed approach has been found to be 96, 92 and 88% for detection of eye blink, left wink and right wink, respectively. The proposed method has also been tested on ZJU dataset where it has given precision, detection accuracy and false alarm rate of values 94.11, 91.2 and 1.54%, respectively. The proposed system has been used and evaluated for performing various mouse analogous functions using eye blinks and winks. It has given an accuracy of 90, 80 and 90% in performing left click, double click, and right click operations, respectively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Krolak A, Strumillo P (2012) Eye-blink detection system for human–computer interaction. Univers Access Inf Soc 11:409–419. https://doi.org/10.1007/s10209-011-0256-6

    Article  Google Scholar 

  2. Divjak M, Bischof H (2009) eye blink based fatigue detection for prevention of computer vision syndrome. In: IAPR conference on machine vision applications, pp 350–353

  3. Danisman T, Bilasco IM, Djeraba C, Ihaddadene N (2010) Drowsy driver detection system using eye blink patterns. In: International conference on machine and web intelligence, pp 230–233

  4. Pan G, Sun L, Wu Z (2007) Eyeblink-based anti-spoofing in face recognition from a generic webcamera. In: 11th IEEE international conference on computer vision (ICCV’07)

  5. Kumar M, Paepcke A, Winograd T (2007) EyePoint?: practical pointing and selection using gaze and keyboard. In: CHI’07 proceedings of the SIGCHI conference on human factors in computing systems, pp 421–430

  6. Singh JV, Prasad G (2015) Enhancing an eye-tracker based human–computer interface with multi-modal accessibility applied for text entry. Int J Comput Appl 130:16–22

    Google Scholar 

  7. Guillamet TPA, Teixidó MTM, Viso AF, Palacín CRJ (2013) Implementation of a robust absolute virtual head mouse combining face detection, template matching and optical flow algorithms. Telecommun Syst 52:1479–1489. https://doi.org/10.1007/s11235-011-9625-y

    Article  Google Scholar 

  8. Huckauf A, Urbina MH (2011) Object selection in gaze controlled systems: what you don’t look at is what you get. ACM Trans Appl Percept 8:1–14. https://doi.org/10.1145/1870076.1870081

    Article  Google Scholar 

  9. Urbina MH, Huckauf A (2010) Alternatives to single character entry and dwell time selection on eye typing. In: Proceedings of the 2010 symposium on eye-tracking research & applications, pp 315–322

  10. Vertegaal R (2008) A Fitts’ law comparison of eye tracking and manual input in the selection of visual targets. In: ICMI’08 the 10th international conference on multimodal interfaces, pp 241–248

  11. Gips J, Olivieri P, Tecce J (1993) Direct control of the computer through electrodes placed around the eyes. In: Proceedings of the fifth international conference on human–computer interaction: applications and case studies, pp 630–635

  12. Kraichan C, Pumrin S (2014) Face and eye tracking for controlling computer functions. In: 2014 11th international conference on electrical engineering/electronics, computers, telecommunication and information technology. https://doi.org/10.1109/ECTICon.2014.6839834

  13. Mackenzie IS, Ashtiani B (2011) BlinkWrite?: efficient text entry using eye blinks. Univers Access Inf Soc 10:69–80. https://doi.org/10.1007/s10209-010-0188-6

    Article  Google Scholar 

  14. Arai K, Mardiyanto R (2010) Eye-based HCI with Full specification of mouse and keyboard using pupil knowledge in the gaze estimation. In: Proceedings—2011 8th international conference on information technology: new generations, ITNG 2011, pp 423–428. https://doi.org/10.1109/ITNG.2011.81

  15. Khilari R (2010) Iris tracking and blink detection for human-computer interaction using a low resolution webcam. In: ICVGIP’10 proceedings of the 7th Indian conference on computer vision, graphics and image processing, pp 456–463

  16. Grauman K, Betke M, Lombardi J et al (2003) Communication via eye blinks and eyebrow raises?: video-based human–computer interfaces. Univers Access Inf Soc 2:359–373

    Article  Google Scholar 

  17. Park KS, Lee KT (1996) Eye controlled human computer interface using the line-of-sight and the intentional blink. Comput Ind Eng 30:463–473

    Article  Google Scholar 

  18. Siriluck W, Kamolphiwong S, Kamolphiwong T (2007) Blink and click. In: Proceedings of the 1st international convention on rehabilitation engineering and assistive technology: in conjunction with 1st Tan Tock Seng Hospital neurorehabilitation meeting, pp 43–46

  19. Huckauf A, Urbina MH, Weimar B (2008) On object selection in gaze controlled environments. J Eye Mov Res 2:1–7

    Google Scholar 

  20. Yang SW, Lin CS, Lin SK, Lee CH (2013) Design of virtual keyboard using blink control method for the severely disabled. Comput Methods Programs Biomed 111:410–418. https://doi.org/10.1016/j.cmpb.2013.04.012

    Article  Google Scholar 

  21. Surakka V, Illi M, Isokoski P (2004) Gazing and frowning as a new human–computer interaction technique. ACM Trans Appl Percept 1:40–56

    Article  Google Scholar 

  22. Gizatdinova Y, Spakov O, Surakka V (2012) Comparison of video-based pointing and selection techniques for hands-free text entry. In: Proceedings of the international working conference on advanced visual interfaces, pp 132–139

  23. Zhao XA, Guestrin ED, Sayenko D et al (2012) Typing with eye-gaze and tooth-clicks. In: Proceedings of ETRA 2012, pp 341–344

  24. Rantanen V, Verho J, Lekkala J, et al (2012) The effect of clicking by smiling on the accuracy of head-mounted gaze tracking. In: Eye tracking research and applications symposium (ETRA), pp 345–348

  25. Usakli AB, Gurkan S, Aloise F et al (2010) On the use of electrooculogram for efficient human computer interfaces. Comput Intell Neurosci 2010:1–5. https://doi.org/10.1155/2010/135629

    Google Scholar 

  26. Pander T (2008) An application of detection function for the eye blinking detection. In: Human–computer systems interaction part of the advances in intelligent and soft-computing book series, pp 181–191

  27. Yamagishi K, Hori J, Miyakawa M (2006) Development of EOG-based communication system controlled by eight-directional eye movements. In: 28th IEEE EMBS annual international conference, pp 2574–2577

  28. Pimplaskar D, Nagmode MS, Borkar A (2013) Real time eye blinking detection and tracking using opencv. Int J Eng Res Appl 3:1780–1787

    Google Scholar 

  29. Missimer E, Betke M (2010) Blink and wink detection for mouse pointer control. In: PETRA’10 proceedings of the 3rd international conference on pervasive technologies related to assistive environments

  30. Su M, Ye C, Lin S et al (2008) An implementation of an eye-blink-based communication aid for people with severe disabilities. In: International conference on audio, language and image processing 2008, pp 351–356

  31. Venkataramanan S, Prabhat P, Choudhury SR, et al (2005) Biomedical instrumentation based on electrooculogram (EOG) signal processing and application to a hospital alarm system. In: Proceedings—2005 international conference on intelligent sensing and information processing, ICISIP’05, pp 535–540

  32. Kumar D, Poole E (2002) Classification of EOG for human computer interface. In: Proceedings of the second joint EMBS/BMES conference, pp 64–67

  33. Lopez-basterretxea A, Mendez-zorrilla A, Zapirain BG (2015) Eye/head tracking technology to improve HCI with iPad applications. IEEE Sens J 15:2244–2264. https://doi.org/10.3390/s150202244

    Article  Google Scholar 

  34. Deepika SS, Murugesan G (2015) A novel approach for human computer interface on eye movements for disabled people. In: 2015 IEEE international conference on electrical, computer and communication technologies (ICECCT 2015)

  35. Varona CMJ, Perales FJ, Salinas I (2014) Design recommendations for camera-based head-controlled interfaces that replace the mouse for motion-impaired users. Univers Access Inf Soc 13:471–482. https://doi.org/10.1007/s10209-013-0326-z

    Article  Google Scholar 

  36. Polatsek P (2013) Eye blink detection. In: Conference of IIT.SRC, pp 1–8

  37. Chareonsuk W, Kanhaun S, Khawkam K, Wongsawang D (2016) Face and eyes mouse for ALS Patients. In: 2016 Fifth ICT international student project conference (ICT-ISPC), pp 1–4

  38. Gantyala S, Godad W, Phadnis N (2016) Controlling mouse events using eye blink. Int J Adv Res Comput Commun Eng 5:754–756. https://doi.org/10.17148/IJARCCE.2016.53182

    Google Scholar 

  39. Pallejà T, Rubión E, Tresanchez M, Fernández A (2008) Using the optical flow to implement a relative virtual mouse controlled by head movements. J Univers Comput Sci 14:3127–3141

    Google Scholar 

  40. Viola P, Jones M (2001) Rapid object detection using a boosted cascade of simple features. In: Proceedings of the 2001 IEEE computer society conference on computer vision and pattern recognition, pp 511–518

  41. Mohammed AA, Anwer SA (2014) Efficient eye blink detection method for disabled- helping domain. Int J Adv Comput Sci Appl 5:202–206

    Google Scholar 

  42. Won OhL, Eui Chul L, Kang Ryoung P (2010) Blink detection robust to various facial poses. J Neurosci Methods 193:356–372. https://doi.org/10.1016/j.jneumeth.2010.08.034

    Article  Google Scholar 

  43. Drutarovsky T, Fogelton A (2014) Eye blink detection using variance of motion vectors. In: ECCV 2014 workshop on computer vision, pp 1–12

Download references

Author information

Authors and Affiliations

  1. I.K. Gujral Punjab Technical University, Jalandhar, India

    Hari Singh

  2. Department of Electronics and Communication Engineering, Beant College of Engineering and Technology, Gurdaspur, India

    Jaswinder Singh

  3. Department of Electronics and Communication Engineering, DAV Institute of Engineering and Technology, Kabir Nagar, Jalandhar, 144008, India

    Hari Singh

Authors
  1. Hari Singh
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jaswinder Singh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hari Singh.

Ethics declarations

Conflict of interest

We have no conflict of interest to declare.

Funding

This research did not receive any specific grant from funding agencies in the public, commercial or not-for-profit sectors.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Singh, H., Singh, J. Real-time eye blink and wink detection for object selection in HCI systems. J Multimodal User Interfaces 12, 55–65 (2018). https://doi.org/10.1007/s12193-018-0261-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12193-018-0261-7

Keywords

Search

Navigation