Recognition of Hand Gestures and Conversion of Voice for Betterment of Deaf and Mute People

  • Shubham Kr. Mishra
  • Sheona Sinha
  • Sourabh Sinha
  • Saurabh BilgaiyanEmail author
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 1046)


Around 5% of people across the globe have difficulty in speaking or are unable to speak. So, to overcome this difficulty, sign language came into the picture. It is a method of non-verbal communication which is usually used by deaf and mute people. Another problem that arises with sign language is that people without hearing or speaking problems do not learn this language. This problem is severe as it creates a barrier between them. To resolve this issue, this paper makes use of computer vision and machine learning along with Convolutional Neural Network. The objective of this paper is to facilitate communication among deaf and mute and other people. For achieving this objective, a system is built to convert hand gestures to voice with gesture understanding and motion capture. This system will be helpful for deaf and mute people as it will increase their communication with other people.


Sign language Gesture recognition Computer vision Image processing Convolutional Neural Network 


  1. 1.
    Bhat, S., Amruthesh, M., Ashik, Das, C., Sujith: Translating Indian sign language to text and voice messages using flex sensors. Int. J. Adv. Res. Comput. Commun. Eng. (4), 430–434 (2015)Google Scholar
  2. 2.
    Corballis, M.C.: The gestural origins of language. WIREs Cogn. Sci. 1, 2–7 (2010)CrossRefGoogle Scholar
  3. 3.
    Xu, J., Gannon, P., Emmorey, K., Smith, J., Braun, A.: Symbolic gestures and spoken language are processed by a common neural system. Proc. Natl. Acad. Sci. U.S.A. 106(49), 20664–20669 (2009)CrossRefGoogle Scholar
  4. 4.
    Chakravorty, P.: What is a signal. IEEE Signal Process. Mag. 35(5), 175–177 (2018)CrossRefGoogle Scholar
  5. 5.
    Tang, J., Rangayyan, R., Yao, J., Yang, Y.: Digital image processing and pattern recognition techniques for detection of cancer. Pattern Recogn. 42(6), 1015–1016 (2009)CrossRefGoogle Scholar
  6. 6.
    Klette, R.: Concise Computer Vision, pp. 1–429. Springer, Heidelberg (2014). Scholar
  7. 7.
    Morris, T.: Computer Vision and Image Processing, pp. 1–320. Palgrave Macmillan. Springer (2014)Google Scholar
  8. 8.
    Garg, P., Aggarwal, N., Sofat, S.: Vision based hand gesture recognition. World Acad. Sci. Eng. Technol. Int. J. Comput. Electr. Autom. Control Inf. Eng. 3(1), 186–191 (2009)Google Scholar
  9. 9.
    Li, Y., Chen, X.: Sign-component-based framework for chinese sign language recognition using accelerometer and sEMG data. IEEE Trans. Biomed. Eng. 59(10), 2695–2704 (2012)CrossRefGoogle Scholar
  10. 10.
    Chen, F., Fu, C., Huang, C.: Hand gesture recognition using a real-time tracking method and hidden Markov models. Image Vis. Comput. 21, 745–758 (2003)CrossRefGoogle Scholar
  11. 11.
    Quek, F.: Towards a vision based hand gesture interface. In: Proceedings of Virtual Reality Software and Technology, pp. 17–31. ACM (1994)Google Scholar
  12. 12.
    Wikipedia: Markov model (2018).
  13. 13.
    Cranenburgh, S.V., Alwoshee, A.: An artificial neural network based approach to investigate travellers’ decision rules. Transp. Res. Part C: Emerg. Technol. 98(1), 152–166 (2019)CrossRefGoogle Scholar
  14. 14.
    Jeanguillaume, C., Colliex, C.: Spectrum image: the next step in EELS digital acquisition and processing. Ultramicroscopy 28(1), 252–257 (1989)CrossRefGoogle Scholar
  15. 15.
    Zhang, W., Itoh, K., et al.: Parallel distributed processing model with local space-invariant and its optical architecture. Appl. Opt. 29(32), 4790–4797 (1990)CrossRefGoogle Scholar
  16. 16.
    Jimenez-Fernandez, V.M., et al.: Exploring the use of two-dimensional piecewise-linear functions as an alternative model for representing and processing grayscale-images. J. Appl. Res. Technol. 14(5), 311–318 (2016)CrossRefGoogle Scholar
  17. 17.
    Ding, L., Goshtasby, A.: On the canny edge detector. Pattern Recogn. 34(3), 721–725 (2001)CrossRefGoogle Scholar
  18. 18.
    Coa, Z., Wei, Z., Zhang, G.: A no-reference sharpness metric based on the notion of relative blur for Gaussian blurred images. J. Vis. Commun. Image Represent. 25(7), 1763–1773 (2014)CrossRefGoogle Scholar
  19. 19.
    Ji, S., Xu, W., Yang, M., Yu, K.: 3D convolutional neural networks for human action recognition. IEEE Trans. Pattern Anal. Mach. Intell. 35(1), 221–231 (2013)CrossRefGoogle Scholar
  20. 20.
    Johnston, L.: What are webcam frame rates? (2018).
  21. 21.
    Shinde, S.S., Autee, R.M., Bhosale, V.K.: Real time two way communication approach for hearing impaired and dumb person based on image processing. In: IEEE International Conference on Computational Intelligence and Computing Research (ICCIC), pp. 1–5. IEEE (2016)Google Scholar
  22. 22.
    Seo, Y., Shin, K.S.: Hierarchical convolutional neural networks for fashion image classification. Expert Syst. Appl. 116, 328–339 (2018)CrossRefGoogle Scholar
  23. 23.
    Shi, D., Zhu, L., Cheng, Z., Li, Z., Zhang, H.: Unsupervised multi-view feature extraction with dynamic graph learning. J. Vis. Commun. Image Represent. 56, 256–264 (2018)CrossRefGoogle Scholar
  24. 24.
    Cai, J., Luo, J., Wang, S., Yang, S.: Feature selection in machine learning: a new perspective. NeuroComputing 300, 70–79 (2018)CrossRefGoogle Scholar
  25. 25.
    Xue, Y., Wang, Q., Yin, X.: A unified approach to sufficient dimension reduction. J. Stat. Plan. Infer. 197, 168–179 (2018)MathSciNetCrossRefGoogle Scholar
  26. 26.
    Di, H., Gao, D.: Gray-level transformation and Canny edge detection for 3D seismic discontinuity enhancement. Comput. Geo Sci. 72, 192–200 (2014)CrossRefGoogle Scholar
  27. 27.
    Gaur, R., Chouhan, V.S.: Classifiers in Image processing. Int. J. Future Revolut. Comput. Sci. Commun. Eng. 3(6), 22–24 (2017)Google Scholar
  28. 28.
    Rehm, M., Bee, N., André, E.: Wave like an egyptian – accelerometer based gesture recognition for culture specific interactions. Br. Comput. Soc. 1, 13–22 (2008)Google Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2019

Authors and Affiliations

  • Shubham Kr. Mishra
    • 1
  • Sheona Sinha
    • 1
  • Sourabh Sinha
    • 1
  • Saurabh Bilgaiyan
    • 1
    Email author
  1. 1.School of Computer EngineeringKIIT Deemed to be UniversityBhubaneswarIndia

Personalised recommendations