Advertisement

Head Movement Mouse Control Using Convolutional Neural Network for People with Disabilities

  • Murat ArslanEmail author
  • Idoko John Bush
  • Rahib H. Abiyev
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 896)

Abstract

In this paper, computer mouse control with head movements and eye blinks was proposed for people with impaired spinal cord injury. The head mouse control is based on finding and predicting eye states and direction of the head. This human computer interface (HCI) is an assistant system for people with physical disabilities who are suffering from motor neuron diseases or severe cerebral palsy. By moving the head to the right, left, up and down moves the mouse pointer and sends mouse button commands using eye blinks. Here, left eye-blink triggers left mouse button, right eye-blink triggers right mouse button and double-eyed sends “holds” command. In this system, eye blink and head movement used same Convolutional Neural Network (CNN) architecture with different number of classes (output). In head movement part, CNN has 5 outputs, (forward, up, down, left and right), in eye-blink part CNN has 2 output either opened or closed. This combined system allows people with down to neck paralyzed, to control computer using head movement and eye blinking. The test results reveal that this system is robust and accurate. This invention allows people with disabilities to use computer with head movements and eye blinks without using any extra hardware.

Keywords

Computer mouse People with disabilities Convolutional Neural Network Computer vision Deep learning Haar cascade 

References

  1. 1.
    Vasa, J.J.: Electronic aids for the disabled and the elderly. Med. Instrument. 16(5), 263–264 (1982)Google Scholar
  2. 2.
    Chen, Y.L., Tang, F.T., Chang, W.H., Wong, M.K., Shih, Y.Y., Kuo, T.S.: The new design of an infrared-controlled human computer interface for the disabled. IEEE Trans. Rehab. Eng. 7, 474–481 (1999)CrossRefGoogle Scholar
  3. 3.
    Computer access solutions for the motor impaired, 12 April 1999. http://www.orin.com/access/
  4. 4.
    King, D.E.: Dlib-ml: a machine learning toolkit. JMLR 10, 1755–1758 (2009)Google Scholar
  5. 5.
    Lecun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998).  https://doi.org/10.1109/5.726791CrossRefGoogle Scholar
  6. 6.
    Hecht-Nielsen, R.: Theory of the backpropagation neural network. Neural Netw. 1, 445 (1988).  https://doi.org/10.1016/0893-6080(88)90469-8CrossRefGoogle Scholar
  7. 7.
    Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., et al.: ImageNet large scale visual recognition challenge. Int. J. Comput. Vis. 115(3), 211–252 (2015).  https://doi.org/10.1007/s11263-015-0816-yMathSciNetCrossRefGoogle Scholar
  8. 8.
    Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional networks. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) Computer Vision – ECCV 2014, ECCV 2014. Lecture Notes in Computer Science, vol. 8689, pp. 818–833. Springer, Cham (2014).  https://doi.org/10.1007/978-3-319-10590-1_53CrossRefGoogle Scholar
  9. 9.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: Proceedings of ICLR 2015, pp. 1–14 (2014)Google Scholar
  10. 10.
    Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1–9 (2015).  https://doi.org/10.1109/cvpr.2015.7298594
  11. 11.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016).  https://doi.org/10.1109/cvpr.2016.90
  12. 12.
    Lawrence, S., Giles, C., Tsoi, A.C., Back, A.: Face recognition: a convolutional neural-network approach. IEEE Trans. Neural Netw. 8(1), 98–113 (1997).  https://doi.org/10.1109/72.554195CrossRefGoogle Scholar
  13. 13.
    Ciresan, D., Meier, U., Gambardella, L., Schmidhuber, J.: Convolutional neural network committees for handwritten character classification. In: Proceedings of 2011 International Conference on Document Analysis and Recognition, pp. 3207–3220 (2011).  https://doi.org/10.1109/icdar.2011.229
  14. 14.
    Simard, P., Steinkraus, D., Platt, J.: Best practices for convolutional neural networks applied to visual document analysis. In: Proceedings of Seventh International Conference on Document Analysis and Recognition, pp. 958–963.  https://doi.org/10.1109/icdar.2003.1227801
  15. 15.
    Jiao, L., Zhang, S., Li, L., Liu, F., Ma, W.: A modified convolutional neural network for face sketch synthesis. Pattern Recogn. 76, 125–136 (2018).  https://doi.org/10.1016/j.patcog.2017.10.025CrossRefGoogle Scholar
  16. 16.
    Chudzik, P., Majumdar, S., Calivá, F., Al-Diri, B., Hunter, A.: Microaneurysm detection using fully convolutional neural networks. Comput. Methods Programs Biomed. 158, 185–192 (2018).  https://doi.org/10.1016/j.cmpb.2018.02.016CrossRefGoogle Scholar
  17. 17.
    Li, J., Feng, J., Kuo, C.: Deep convolutional neural network for latent fingerprint enhancement. Sig. Process. Image Commun. 60, 52–63 (2018).  https://doi.org/10.1016/j.image.2017.08.010CrossRefGoogle Scholar
  18. 18.
    Hussain, S., Anwar, S., Majid, M.: Segmentation of glioma tumors in brain using deep convolutional neural network. Neurocomputing 282, 248–261 (2018).  https://doi.org/10.1016/j.neucom.2017.12.032CrossRefGoogle Scholar
  19. 19.
    Baldominos, A., Saez, Y., Isasi, P.: Evolutionary convolutional neural networks: an application to handwriting recognition. Neurocomputing 283, 38–52 (2018).  https://doi.org/10.1016/j.neucom.2017.12.049CrossRefGoogle Scholar
  20. 20.
    Ferreira, A., Giraldi, G.: Convolutional neural network approaches to granite tiles classification. Expert Syst. Appl. 84, 1–11 (2017).  https://doi.org/10.1016/j.eswa.2017.04.053CrossRefGoogle Scholar
  21. 21.
    Abiyev, R., Ma’aitah, M.K.S.: Deep convolutional neural networks for chest diseases detection. J. Healthc. Eng. 2018 (2018)Google Scholar
  22. 22.
    Wachinger, C., Reuter, M., Klein, T.: DeepNAT: deep convolutional neural network for segmenting neuroanatomy. Neuroimage 170, 434–445 (2018).  https://doi.org/10.1016/j.neuroimage.2017.02.035CrossRefGoogle Scholar
  23. 23.
    Viola, P., Jones, M.: Robust real-time face detection. Int. J. Comput. Vis. 57(2), 137–154 (2004)CrossRefGoogle Scholar
  24. 24.
    Kingma, D.P., Ba, J.L.: ADAM: a method for stochastic optimization. In: ICLR 2015 (2015)Google Scholar
  25. 25.
    TensorFlow. https://www.tensorflow.org/. Accessed 19 June 2018
  26. 26.
    Song, F., Tan, X., Liu, X., Chen, S.: Eyes closeness detection from still images with multi-scale histograms of principal oriented gradients. Pattern Recogn. 47(9), 2825–2838 (2014)CrossRefGoogle Scholar
  27. 27.
    Abiyev, R.H., Altunkaya, K.: Neural network based biometric personal identification using fast iris segmentation. Int. J. Control Autom. Syst. 7(1), 17–23 (2009)CrossRefGoogle Scholar
  28. 28.
    Abiyev, R.H., Abizade, S.: Diagnosing Parkinson’s diseases using fuzzy neural system. Comput. Math. Methods Med. 2016 (2016). Article ID 1267919MathSciNetCrossRefGoogle Scholar
  29. 29.
    Abiyev, R.H., Kilic, K.: Adaptive Iris Segmentation. Lecture Notes in Computer Sciences. Springer/CS Press, Heidelberg (2009)CrossRefGoogle Scholar
  30. 30.
    Abiyev, R., Altunkaya, K.: Personal iris recognition using neural networks. Int. J. Secur. Appl. 2(2), 41–50 (2008)Google Scholar
  31. 31.
    Abiyev, R., Altunkaya, K.: Neural Network Based Biometric Personal Identification. Lecture Notes in Computer Sciences. Springer/CS Press, Heidelberg (2007)CrossRefGoogle Scholar
  32. 32.
    Helwan, A., Ozsahin, D.U., Abiyev, R., John, B.: One-year survival prediction of myocardial infarction. (IJACSA). Int. J. Adv. Comput. Sci. Appl. 8, 173–178 (2017)Google Scholar
  33. 33.
    Idoko, J.B., Abiyev, R.H., Ma’aitah, M.K.S.: Intelligent machine learning algorithms for colour segmentation. WSEAS Trans. Sig. Process. 13, 232–240 (2017)Google Scholar
  34. 34.
    Idoko, J.B., Abiyev, R., Ma’aitah, M.K.S., Hamit, A.: Integrated artificial intelligence algorithm for skin detection. In: International Conference Applied Mathematics, Computational Science and Systems Engineering (AMCSE 2017), ITM Web of Conferences, vol. 16, p. 02004 (2018)Google Scholar
  35. 35.
    Ma’aitah, M.K.S., Abiyev, R., Idoko, J.B.: Intelligent classification of liver disorder using fuzzy neural system. Int. J. Adv. Comput. Sci. Appl. (IJACSA) 8, 25–31 (2017)Google Scholar
  36. 36.
    Helwan, A., Idoko, J.B., Abiyev, R.H.: Machine learning techniques for classification of breast tissue. In: 9th International Conference on Theory and Application of Soft Computing, Computing with Words and Perception, ICSCCW 2017. Procedia Computer Science, vol. 120, pp. 402–410 (2017)CrossRefGoogle Scholar
  37. 37.
    Abiyev, R.H., Arslan, M., Gunsel, I., Cagman, A.: Robot pathfinding using vision based obstacle detection. In: IEEE International Conference on Cybernetics, pp. 1–6 (2017)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Murat Arslan
    • 1
    Email author
  • Idoko John Bush
    • 1
  • Rahib H. Abiyev
    • 1
  1. 1.Applied Artificial Intelligence Research Centre, Department of Computer EngineeringNear East UniversityMersin-10Turkey

Personalised recommendations