Head Movement Mouse Control Using Convolutional Neural Network for People with Disabilities

  • Murat ArslanEmail author
  • Idoko John Bush
  • Rahib H. Abiyev
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 896)


In this paper, computer mouse control with head movements and eye blinks was proposed for people with impaired spinal cord injury. The head mouse control is based on finding and predicting eye states and direction of the head. This human computer interface (HCI) is an assistant system for people with physical disabilities who are suffering from motor neuron diseases or severe cerebral palsy. By moving the head to the right, left, up and down moves the mouse pointer and sends mouse button commands using eye blinks. Here, left eye-blink triggers left mouse button, right eye-blink triggers right mouse button and double-eyed sends “holds” command. In this system, eye blink and head movement used same Convolutional Neural Network (CNN) architecture with different number of classes (output). In head movement part, CNN has 5 outputs, (forward, up, down, left and right), in eye-blink part CNN has 2 output either opened or closed. This combined system allows people with down to neck paralyzed, to control computer using head movement and eye blinking. The test results reveal that this system is robust and accurate. This invention allows people with disabilities to use computer with head movements and eye blinks without using any extra hardware.


Computer mouse People with disabilities Convolutional Neural Network Computer vision Deep learning Haar cascade 


  1. 1.
    Vasa, J.J.: Electronic aids for the disabled and the elderly. Med. Instrument. 16(5), 263–264 (1982)Google Scholar
  2. 2.
    Chen, Y.L., Tang, F.T., Chang, W.H., Wong, M.K., Shih, Y.Y., Kuo, T.S.: The new design of an infrared-controlled human computer interface for the disabled. IEEE Trans. Rehab. Eng. 7, 474–481 (1999)CrossRefGoogle Scholar
  3. 3.
    Computer access solutions for the motor impaired, 12 April 1999.
  4. 4.
    King, D.E.: Dlib-ml: a machine learning toolkit. JMLR 10, 1755–1758 (2009)Google Scholar
  5. 5.
    Lecun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998). Scholar
  6. 6.
    Hecht-Nielsen, R.: Theory of the backpropagation neural network. Neural Netw. 1, 445 (1988). Scholar
  7. 7.
    Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., et al.: ImageNet large scale visual recognition challenge. Int. J. Comput. Vis. 115(3), 211–252 (2015). Scholar
  8. 8.
    Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional networks. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) Computer Vision – ECCV 2014, ECCV 2014. Lecture Notes in Computer Science, vol. 8689, pp. 818–833. Springer, Cham (2014). Scholar
  9. 9.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: Proceedings of ICLR 2015, pp. 1–14 (2014)Google Scholar
  10. 10.
    Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1–9 (2015).
  11. 11.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016).
  12. 12.
    Lawrence, S., Giles, C., Tsoi, A.C., Back, A.: Face recognition: a convolutional neural-network approach. IEEE Trans. Neural Netw. 8(1), 98–113 (1997). Scholar
  13. 13.
    Ciresan, D., Meier, U., Gambardella, L., Schmidhuber, J.: Convolutional neural network committees for handwritten character classification. In: Proceedings of 2011 International Conference on Document Analysis and Recognition, pp. 3207–3220 (2011).
  14. 14.
    Simard, P., Steinkraus, D., Platt, J.: Best practices for convolutional neural networks applied to visual document analysis. In: Proceedings of Seventh International Conference on Document Analysis and Recognition, pp. 958–963.
  15. 15.
    Jiao, L., Zhang, S., Li, L., Liu, F., Ma, W.: A modified convolutional neural network for face sketch synthesis. Pattern Recogn. 76, 125–136 (2018). Scholar
  16. 16.
    Chudzik, P., Majumdar, S., Calivá, F., Al-Diri, B., Hunter, A.: Microaneurysm detection using fully convolutional neural networks. Comput. Methods Programs Biomed. 158, 185–192 (2018). Scholar
  17. 17.
    Li, J., Feng, J., Kuo, C.: Deep convolutional neural network for latent fingerprint enhancement. Sig. Process. Image Commun. 60, 52–63 (2018). Scholar
  18. 18.
    Hussain, S., Anwar, S., Majid, M.: Segmentation of glioma tumors in brain using deep convolutional neural network. Neurocomputing 282, 248–261 (2018). Scholar
  19. 19.
    Baldominos, A., Saez, Y., Isasi, P.: Evolutionary convolutional neural networks: an application to handwriting recognition. Neurocomputing 283, 38–52 (2018). Scholar
  20. 20.
    Ferreira, A., Giraldi, G.: Convolutional neural network approaches to granite tiles classification. Expert Syst. Appl. 84, 1–11 (2017). Scholar
  21. 21.
    Abiyev, R., Ma’aitah, M.K.S.: Deep convolutional neural networks for chest diseases detection. J. Healthc. Eng. 2018 (2018)Google Scholar
  22. 22.
    Wachinger, C., Reuter, M., Klein, T.: DeepNAT: deep convolutional neural network for segmenting neuroanatomy. Neuroimage 170, 434–445 (2018). Scholar
  23. 23.
    Viola, P., Jones, M.: Robust real-time face detection. Int. J. Comput. Vis. 57(2), 137–154 (2004)CrossRefGoogle Scholar
  24. 24.
    Kingma, D.P., Ba, J.L.: ADAM: a method for stochastic optimization. In: ICLR 2015 (2015)Google Scholar
  25. 25.
    TensorFlow. Accessed 19 June 2018
  26. 26.
    Song, F., Tan, X., Liu, X., Chen, S.: Eyes closeness detection from still images with multi-scale histograms of principal oriented gradients. Pattern Recogn. 47(9), 2825–2838 (2014)CrossRefGoogle Scholar
  27. 27.
    Abiyev, R.H., Altunkaya, K.: Neural network based biometric personal identification using fast iris segmentation. Int. J. Control Autom. Syst. 7(1), 17–23 (2009)CrossRefGoogle Scholar
  28. 28.
    Abiyev, R.H., Abizade, S.: Diagnosing Parkinson’s diseases using fuzzy neural system. Comput. Math. Methods Med. 2016 (2016). Article ID 1267919MathSciNetCrossRefGoogle Scholar
  29. 29.
    Abiyev, R.H., Kilic, K.: Adaptive Iris Segmentation. Lecture Notes in Computer Sciences. Springer/CS Press, Heidelberg (2009)CrossRefGoogle Scholar
  30. 30.
    Abiyev, R., Altunkaya, K.: Personal iris recognition using neural networks. Int. J. Secur. Appl. 2(2), 41–50 (2008)Google Scholar
  31. 31.
    Abiyev, R., Altunkaya, K.: Neural Network Based Biometric Personal Identification. Lecture Notes in Computer Sciences. Springer/CS Press, Heidelberg (2007)CrossRefGoogle Scholar
  32. 32.
    Helwan, A., Ozsahin, D.U., Abiyev, R., John, B.: One-year survival prediction of myocardial infarction. (IJACSA). Int. J. Adv. Comput. Sci. Appl. 8, 173–178 (2017)Google Scholar
  33. 33.
    Idoko, J.B., Abiyev, R.H., Ma’aitah, M.K.S.: Intelligent machine learning algorithms for colour segmentation. WSEAS Trans. Sig. Process. 13, 232–240 (2017)Google Scholar
  34. 34.
    Idoko, J.B., Abiyev, R., Ma’aitah, M.K.S., Hamit, A.: Integrated artificial intelligence algorithm for skin detection. In: International Conference Applied Mathematics, Computational Science and Systems Engineering (AMCSE 2017), ITM Web of Conferences, vol. 16, p. 02004 (2018)Google Scholar
  35. 35.
    Ma’aitah, M.K.S., Abiyev, R., Idoko, J.B.: Intelligent classification of liver disorder using fuzzy neural system. Int. J. Adv. Comput. Sci. Appl. (IJACSA) 8, 25–31 (2017)Google Scholar
  36. 36.
    Helwan, A., Idoko, J.B., Abiyev, R.H.: Machine learning techniques for classification of breast tissue. In: 9th International Conference on Theory and Application of Soft Computing, Computing with Words and Perception, ICSCCW 2017. Procedia Computer Science, vol. 120, pp. 402–410 (2017)CrossRefGoogle Scholar
  37. 37.
    Abiyev, R.H., Arslan, M., Gunsel, I., Cagman, A.: Robot pathfinding using vision based obstacle detection. In: IEEE International Conference on Cybernetics, pp. 1–6 (2017)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Murat Arslan
    • 1
    Email author
  • Idoko John Bush
    • 1
  • Rahib H. Abiyev
    • 1
  1. 1.Applied Artificial Intelligence Research Centre, Department of Computer EngineeringNear East UniversityMersin-10Turkey

Personalised recommendations