Hand Gesture Recognition for Human Computer Interaction and Its Applications in Virtual Reality

  • Sarthak Gupta
  • Siddhant Bagga
  • Deepak Kumar SharmaEmail author
Part of the Studies in Computational Intelligence book series (SCI, volume 875)


Computers are emerging as the most utilitarian products in the human society and therefore the interaction between humans and computers will have a very significant influence in the society. As a result, enormous amount of efforts are being made to augment the research in the domain of human computer interaction to develop more efficient and effective techniques for the purpose of reducing the barrier of humans and computers. The primary objective is to develop a conducive environment in which there is feasibility of very natural interaction between humans and computers. In order to achieve this goal, gestures play a very pivotal role and are the core area of research in this domain. Hand gesture recognition is a significant component of virtual Reality finds applications in numerous fields including video games, cinema, robotics, education, marketing, etc. Virtual reality also caters to a variety of healthcare applications involving the procedures used in surgical operations including remote surgery, augmented surgery, software emulation of the surgeries prior to actual surgeries, therapies, training in the medical education, medical data visualization and much more. A lot of tools and techniques have. Been developed to cater to the development of the such virtual environments. Gesture recognition signifies the method of keeping track of gestures of humans, to representing and converting the gestures to meaningful signals. Contact based and vision based devices are used for creating and implementing these systems of recognition effectively. The chapter begins with the introduction of hand gesture recognition and the process of carrying out hand gesture recognition. Further, the latest research which is being in carried out in the domain of hand gesture recognition is described. It is followed by the details of applications of virtual reality and hand gesture recognition in the field of healthcare. Then, various techniques which are applied in hand gesture recognition are described. Finally, the challenges in the field of hand gesture recognition have been explained.


Artificial intelligence Virtual reality Hand gesture recognition Human computer interaction Healthcare Representations Recognition Natural interfaces 


  1. 1.
    Sinha, G., Shahi, R., & Shankar, M. (2010). Human Computer Interaction. 2010 3rd International Conference on Emerging Trends in Engineering and Technology.Google Scholar
  2. 2.
    Chakraborty, B. K., Sarma, D., Bhuyan, M. K., & MacDorman, K. F. (2018). Review of constraints on vision-based gesture recognition for human–computer interaction. IET Computer Vision, 12(1), 3–15.CrossRefGoogle Scholar
  3. 3.
    Jaimes, A., & Sebe, N. (2007). Multimodal human-computer interaction: A survey. Computer Vision and Image Understanding, 108(1–2), 116–134.CrossRefGoogle Scholar
  4. 4.
    Chapanis, A. (1965). Man machine engineering. Belmont: Wadsworth.Google Scholar
  5. 5.
    Norman, D. (1986). Cognitive Engineering. In D. Norman & S. Draper (Eds.), User centered design: New perspective on human-computer interaction. Hillsdale: Lawrence Erlbaum.CrossRefGoogle Scholar
  6. 6.
    Picard, R. W. (1997). Affective computing. Cambridge: MIT Press.Google Scholar
  7. 7.
    Han, Y. (2010). A low-cost visual motion data glove as an input device to interpret human hand gestures. IEEE Transactions on Consumer Electronics, 56(2), 501–509.CrossRefGoogle Scholar
  8. 8.
    Choudhury, A., Talukdar, A. K., & Sarma, K. K. (2014). A Conditional Random Field Based Indian Sign Language Recognition System under Complex Background. 2014 Fourth International Conference on Communication Systems and Network Technologies.Google Scholar
  9. 9.
    Habili, N., Lim, C. C., & Moini, A. (2004). Segmentation of the face and hands in sign language video sequences using color and motion cues. IEEE Transactions on Circuits and Systems for Video Technology, 14(8), 1086–1097.CrossRefGoogle Scholar
  10. 10.
    Iqbal, J., Ul Haq, A., & Wali, S. (2015). Moving target detection and tracking.Google Scholar
  11. 11.
    Choudhury, A., Talukdar, A., Sarma, K. (2014). A novel hand segmentation method for multiple-hand gesture recognition system under complex background. In 2014 International Conference on Signal Processing and Integrated Networks, SPIN 2014.
  12. 12.
    Stergiopoulou, E., & Papamarkos, N. (2009). Hand gesture recognition using a neural network shape fitting technique. Engineering Applications of Artificial Intelligence, 22(8), 1141–1158.CrossRefGoogle Scholar
  13. 13.
    Malima, A., Ozgur, E., & Cetin, M. (n.d.). A fast algorithm for vision-based hand gesture recognition for robot control. In 2006 IEEE 14th Signal Processing and Communications Applications.
  14. 14.
    Hasan, M. M., & Mishra, P. K. (2011). HSV brightness factor matching for gesture recognition system. International Journal of Image Processing (IJIP), 4(5), 456–467.Google Scholar
  15. 15.
    Chang, C. C., Chen, J. J., Tai, W., & Han, C. C. (2006). New approach for static gesture recognition. Journal of Information Science and Engineering, 22, 1047–1057.Google Scholar
  16. 16.
    Just, A. (2006). Two-handed gestures for human–computer interaction. Ph.D. thesis.Google Scholar
  17. 17.
    Parvini, F., & Shahabi, C. (2007). An algorithmic approach for static and dynamic gesture recognition utilising mechanical and biomechanical characteristics. International Journal of Bioinformatics Research and Applications, 3(1), 4.CrossRefGoogle Scholar
  18. 18.
    Dardas, N. H., & Georganas, N. D. (2011). Real-time hand gesture detection and recognition using bag-of-features and support vector machine techniques. IEEE Transactions on Instrumentation and Measurement, 60(11), 3592–3607.CrossRefGoogle Scholar
  19. 19.
    Nagi, J., Ducatelle, F., Di Caro, G. A., Ciresan, D., Meier, U., Giusti, A., et al. (2011). Max-pooling convolutional neural networks for vision-based hand gesture recognition. In 2011 IEEE International Conference on Signal and Image Processing Applications (ICSIPA).Google Scholar
  20. 20.
    Panwar, M. (2012). Hand gesture recognition based on shape parameters. In 2012 International Conference on Computing, Communication and Applications.Google Scholar
  21. 21.
    Ohn-Bar, E., & Trivedi, M. M. (2014). Hand gesture recognition in real time for automotive interfaces: A multimodal vision-based approach and evaluations. IEEE Transactions on Intelligent Transportation System, 15(6), 2368–2377.CrossRefGoogle Scholar
  22. 22.
    Suk, H.-I., Sin, B.-K., & Lee, S.-W. (2010). Hand gesture recognition based on dynamic Bayesian network framework. Pattern Recognition, 43(9), 3059–3072.CrossRefGoogle Scholar
  23. 23.
    Shen, X., Hua, G., Williams, L., & Wu, Y. (2012). Dynamic hand gesture recognition: An exemplar-based approach from motion divergence fields. Image and Vision Computing, 30(3), 227–235. Scholar
  24. 24.
    Padam Priyal, S., & Bora, P. K. (2013). A robust static hand gesture recognition system using geometry based normalizations and Krawtchouk moments. Pattern Recognition, 46(8), 2202–2219. Scholar
  25. 25.
    Hoffman, H., & Vu, D. (1997). Virtual reality: teaching tool of the twenty-first century? Academic Medicine: Journal of the Association of American Medical Colleges, 72(12), 1076–1081.CrossRefGoogle Scholar
  26. 26.
    Gallagher, A. G., Ritter, E. M., Champion, H., Higgins, G., Fried, M. P., Moses, G., et al. (2005). Virtual reality simulation for the operating room: Proficiency-based training as a paradigm shift in surgical skills training. Annals of Surgery, 241(2), 364.CrossRefGoogle Scholar
  27. 27.
    Aggarwal, R., Ward, J., Balasundaram, I., Sains, P., Athanasiou, T., & Darzi, A. (2007). Proving the effectiveness of virtual reality simulation for training in laparoscopic surgery. Annals of Surgery, 246(5), 771–779.CrossRefGoogle Scholar
  28. 28.
    Satava, R. M. (1993). Virtual reality surgical simulator. Surgical Endoscopy, 7(3), 203–205.CrossRefGoogle Scholar
  29. 29.
    Liu, J. Q., Fujii, R., Tateyama, T., Iwamoto, Y., & Chen, Y. W. (2017). Kinect-based gesture recognition for touchless visualization of medical images. International Journal of Computer and Electrical Engineering, 9(2), 421–429.CrossRefGoogle Scholar
  30. 30.
    Krapichler, C., Haubner, M., Engelbrecht, R., & Englmeier, K. H. (1998). VR interaction techniques for medical imaging applications. Computer Methods and Programs in Biomedicine, 56(1), 65–74.CrossRefGoogle Scholar
  31. 31.
    Khan, R. Z., & Ibraheem, N. A. (2012). Comparative study of hand gesture recognition system. In Proceedings of International Conference of Advanced Computer Science & Information Technology in Computer Science & Information Technology (CS & IT) (Vol. 2, No. 3, pp. 203–213).Google Scholar
  32. 32.
    Rautaray, S. S., & Agrawal, A. (2015). Vision based hand gesture recognition for human computer interaction: A survey. Artificial Intelligence Review, 43(1), 1–54.CrossRefGoogle Scholar
  33. 33.
    Rehg, J. M., & Kanade, T. (1994, November). Digiteyes: Vision-based hand tracking for human-computer interaction. In Proceedings of the 1994 IEEE Workshop on Motion of Non-rigid and Articulated Objects, 1994 (pp. 16–22). IEEE.Google Scholar
  34. 34.
    Ramesh, V. (2003, October). Background modeling and subtraction of dynamic scenes. In Proceedings. Ninth IEEE International Conference on Computer Vision, 2003 (pp. 1305–1312). IEEE.Google Scholar
  35. 35.
    Zivkovic, Z. (2004, August). Improved adaptive Gaussian mixture model for background subtraction. In Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004 (Vol. 2, pp. 28–31). IEEE.Google Scholar
  36. 36.
    Schapire, R. E. (2003). The boosting approach to machine learning: An overview. In Nonlinear estimation and classification (pp. 149–171). New York, NY: Springer.CrossRefGoogle Scholar
  37. 37.
    Kalman, R. E. (1960). A new approach to linear filtering and prediction problems. Journal of Basic Engineering, 82(1), 35–45.MathSciNetCrossRefGoogle Scholar
  38. 38.
    Stenger, B., Mendonça, P. R., & Cipolla, R. (2001, September). Model-based hand tracking using an unscented Kalman filter. In BMVC (Vol. 1, pp. 63–72).Google Scholar
  39. 39.
    Isard, M., & Blake, A. (1996, April). Contour tracking by stochastic propagation of conditional density. In European Conference on Computer Vision (pp. 343–356). Berlin, Heidelberg: Springer.Google Scholar
  40. 40.
    Shan, C., Tan, T., & Wei, Y. (2007). Real-time hand tracking using a mean shift embedded particle filter. Pattern Recognition, 40(7), 1958–1970.CrossRefGoogle Scholar
  41. 41.
    Stenger, B., Thayananthan, A., Torr, P. H., & Cipolla, R. (2006). Model-based hand tracking using a hierarchical bayesian filter. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(9), 1372–1384.CrossRefGoogle Scholar
  42. 42.
    Nadgeri, S. M., Sawarkar, S. D., & Gawande, A. D. (2010, November). Hand gesture recognition using CAMSHIFT algorithm. In 2010 3rd International Conference on Emerging Trends in Engineering and Technology (ICETET) (pp. 37–41). IEEE.Google Scholar
  43. 43.
    Peng, J. C., Gu, L. Z., & Su, J. B. (2006). The hand tracking for humanoid robot using Camshift algorithm and Kalman filter. Journal-Shanghai Jiaotong University-Chinese Edition, 40(7), 1161.Google Scholar
  44. 44.
    Luo, Y., Li, L., Zhang, B. S., & Yang, H. M. (2009). Video hand tracking algorithm based on hybrid Camshift and Kalman filter. Application Research of Computers, 26(3), 1163–1165.Google Scholar
  45. 45.
    Li, Y. (2012, June). Hand gesture recognition using Kinect. In 2012 IEEE 3rd International Conference on Software Engineering and Service Science (ICSESS) (pp. 196–199). IEEE.Google Scholar
  46. 46.
    Dudani, S. A. (1976). The distance-weighted k-nearest-neighbor rule. IEEE Transactions on Systems, Man, and Cybernetics, 4, 325–327.CrossRefGoogle Scholar
  47. 47.
    Keller, J. M., Gray, M. R., & Givens, J. A. (1985). A fuzzy k-nearest neighbor algorithm. IEEE Transactions on Systems, Man, and Cybernetics, 4, 580–585.CrossRefGoogle Scholar
  48. 48.
    Kollorz, E., Penne, J., Hornegger, J., & Barke, A. (2008). Gesture recognition with a time-of-flight camera. International Journal of Intelligent Systems Technologies and Applications, 5(3), 334.CrossRefGoogle Scholar
  49. 49.
    Chen, Y. T., & Tseng, K. T. (2007, September). Multiple-angle hand gesture recognition by fusing SVM classifiers. In IEEE International Conference on Automation Science and Engineering, 2007. CASE 2007 (pp. 527–530). IEEE.Google Scholar
  50. 50.
    Dardas, N., Chen, Q., Georganas, N. D., & Petriu, E. M. (2010, October). Hand gesture recognition using bag-of-features and multi-class support vector machine. In 2010 IEEE International Symposium on Haptic Audio-Visual Environments and Games (HAVE) (pp. 1–5). IEEE.Google Scholar
  51. 51.
    Chen, F. S., Fu, C. M., & Huang, C. L. (2003). Hand gesture recognition using a real-time tracking method and hidden Markov models. Image and Vision Computing, 21(8), 745–758.CrossRefGoogle Scholar
  52. 52.
    Elmezain, M., Al-Hamadi, A., Appenrodt, J., & Michaelis, B. (2009). A hidden markov model-based isolated and meaningful hand gesture recognition. International Journal of Electrical, Computer, and Systems Engineering, 3(3), 156–163.Google Scholar
  53. 53.
    Waibel, A., Hanazawa, T., Hinton, G., Shikano, K., & Lang, K. J. (1990). Phoneme recognition using time-delay neural networks. In Readings in speech recognition (pp. 393–404).Google Scholar
  54. 54.
    Hong, P., Turk, M., & Huang, T. S. (2000). Constructing finite state machines for fast gesture recognition. In Proceedings 15th International Conference on Pattern Recognition, 2000 (Vol. 3, pp. 691–694). IEEE.Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Sarthak Gupta
    • 1
  • Siddhant Bagga
    • 1
  • Deepak Kumar Sharma
    • 1
    Email author
  1. 1.Department of Information TechnologyNetaji Subhas University of Technology (Formerly Netaji Subhas Institute of Technology)New DelhiIndia

Personalised recommendations