Development of a hierarchical dynamic keyboard character recognition system using trajectory features and scale-invariant holistic modeling of characters

Abstract

A robust, intuitive, effortless, and novel dynamic hand gesture based virtual keyboard system is developed in this study. Firstly, a new hierarchical approach is applied, which, based on self-coarticulation and position features, effectively sub-groups a large gesture vocabulary. Additionally, new trajectory features are proposed which shall extract the local structural statistics of the gestures. All state-of-art models are based on temporal trajectory features which are based on the frame-wise 2D sequential path it followed. Due to this, the trajectory features are path dependent and vulnerable to trajectory noises or any other variations in pattern, speed, or scale. In contrast to this, an image-based approach of gesture recognition has been proposed in this study, which is independent of the sequential gesturing path of the gesture. Since the image-models (a holistic view) are not obtained frame-wise, unlike existing image-models, they are pattern, speed, and scale invariant in nature and also immune to trajectory distortions. To this end, image-based features and significant trajectory features are fused to develop a hybrid hierarchical classification model which exhibits an exceptional increase in accuracy by 3.9% as compared to baseline non-hierarchical trajectory based model using an Artificial neural network (ANN). Classification models such as Voronoi diagram based classifier (VDBC) and neuro-fuzzy (NF) classifier have also been explored and displayed motivating performance. Reduction in misclassification has been observed for gestures such as ‘(and)’, ‘{and}’, ‘0 and O’, ‘Z and 2’. The present system can also identify any static/dynamic imposters present in the gesture environment.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

References

  1. Baig F, Beg S, Khan MF, Nawaz SJ (2015) A method to control home appliances based on writing commands over the air. J Control Autom Elect Syst 26(4):421–429. https://doi.org/10.1007/s40313-015-0184-4

    Article  Google Scholar 

  2. Bhuyan MK, Ghosh D, Bora PK (2006) Hand motion tracking and trajectory matching for dynamic hand gesture recognition. J Exp Theor Artif Intell 18(4):435–447. https://doi.org/10.1080/09528130600975931

    Article  Google Scholar 

  3. Bhuyan MK, Kumar DA, MacDorman KF, Iwahori Y (2014) A novel set of features for continuous hand gesture recognition. J Multimodal User Interfaces 8(4):333–343. https://doi.org/10.1007/s12193-014-0165-0

    Article  Google Scholar 

  4. Bobick AF, Davis JW (2001) The recognition of human movement using temporal templates. IEEE Trans Pattern Anal Mach Intell 23(3):257–267

    Article  Google Scholar 

  5. Bretzner L, Laptev I, Lindeberg T (2002) Hand gesture recognition using multi-scale colour features, hierarchical models and particle filtering. In: Automatic face and gesture recognition, 2002. Proceedings. Fifth IEEE international conference on, IEEE. pp 423–428. https://doi.org/10.1109/AFGR.2002.1004190

  6. Carlsson S, Sullivan J (2001) Action recognition by shape matching to key frames. In: Workshop on models versus exemplars in computer vision, vol 1, p 18

  7. Carreira M, Ting KLH, Csobanka P, Gonçalves D (2017) Evaluation of in-air hand gestures interaction for older people. Univ Access Inf Soc 16(3):561–580. https://doi.org/10.1007/s10209-016-0483-y

    Article  Google Scholar 

  8. Cetisli B (2010) Development of an adaptive neuro-fuzzy classifier using linguistic hedges: part 1. Expert Syst Appl 37(8):6093–6101. https://doi.org/10.1016/j.eswa.2010.02.108

    Article  Google Scholar 

  9. Chang CL (1974) Finding prototypes for nearest neighbor classifiers. IEEE Trans Comput 100(11):1179–1184. https://doi.org/10.1109/T-C.1974.223827

    Article  MATH  Google Scholar 

  10. Choi H, Park H (2014) A hierarchical structure for gesture recognition using RGB-D sensor. In: Proceedings of the second international conference on Human-agent interaction, ACM, pp 265–268. https://doi.org/10.1145/2658861.2658938

  11. Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: Computer vision and pattern recognition, IEEE computer society conference, vol 1, pp 886–893. https://doi.org/10.1109/CVPR.2005.177

  12. Deepthi DR, Eswaran K (2010) A new hierarchical pattern recognition method using mirroring neural networks. Int J Comput Appl 1(12):70–78

    Google Scholar 

  13. Douglas DH, Peucker TK (2011) Algorithms for the reduction of the number of points required to represent a digitized line or its caricature. In: Classics in cartography: reflections on influential articles from cartographica. Wiley Online Library, Hoboken, pp 15–28. https://doi.org/10.1002/9780470669488.ch2

    Google Scholar 

  14. Elmezain M, Al-Hamadi A, Krell G, El-Etriby S, Michaelis B (2007) Gesture recognition for alphabets from hand motion trajectory using hidden markov models. In: 2007 IEEE international symposium on signal processing and information technology, pp 1192–1197. https://doi.org/10.1109/ISSPIT.2007.4458209

  15. Fang G, Gao W, Zhao D (2003) Large vocabulary sign language recognition based on hierarchical decision trees. In: Proceedings of the 5th international conference on multimodal interfaces, ACM. pp 125–131. https://doi.org/10.1109/TSMCA.2004.824852

    Article  Google Scholar 

  16. Kobayashi T, Otsu N (2008) Image feature extraction using gradient local auto-correlations. In: European conference on computer vision. Springer, Berlin, Heidelberg, pp 346–358. https://doi.org/10.1007/978-3-540-88682-2_27

    Google Scholar 

  17. Kundu AS, Mazumder O, Lenka PK, Bhaumik S (2017) Hand gesture recognition based omnidirectional wheelchair control using IMU and EMG sensors. J Intell Robot Syst. https://doi.org/10.1007/s10846-017-0725-0

    Article  Google Scholar 

  18. Li C, Xie C, Zhang B, Chen C, Han J (2018) Deep Fisher discriminant learning for mobile hand gesture recognition. Pattern Recognit 77:276–288. https://doi.org/10.1016/j.patcog.2017.12.023

    Article  Google Scholar 

  19. Liu F, Zeng W, Yuan C, Wang Q, Wang Y, Lu B (2018) Trajectory-based hand gesture recognition using kinect via deterministic learning. In: Chinese control conference. https://doi.org/10.23919/ChiCC.2018.8482621

  20. Misra S, Laskar RH (2018) Approach toward extraction of sparse texture features and their application in robust two-level bare-hand detection. J Electron Imaging 27(5):051209. https://doi.org/10.1117/1.JEI.27.5.051209

    Article  Google Scholar 

  21. Misra S, Singha J, Laskar RH (2018) Vision-based hand gesture recognition of alphabets, numbers, arithmetic operators and ASCII characters in order to develop a virtual text-entry interface system. Neural Comput Appl 29(8):117–135. https://doi.org/10.1007/s00521-017-2838-6

    Article  Google Scholar 

  22. Moschetti A, Fiorini L, Esposito D, Dario P, Cavallo F (2017) Toward an unsupervised approach for daily gesture recognition in assisted living applications. IEEE Sens J 17(24):8395–8403. https://doi.org/10.1109/JSEN.2017.2764323

    Article  Google Scholar 

  23. Ogale A, Karapurkar A, Guerra-Filho G, Aloimonos Y (2004) View-invariant identification of pose sequences for action recognition. In: Video analysis and content extraction workshop (VACE), p 14

  24. Pang Y, Yuan Y, Li X, Pan J (2011) Efficient R-HOG human detection. Sig Process 91(4):773–781. https://doi.org/10.1016/j.sigpro.2010.08.010

    Article  MATH  Google Scholar 

  25. Quesada L, López G, Guerrero L (2017) Automatic recognition of the American sign language fingerspelling alphabet to assist people living with speech or hearing impairments. J Ambient Intell Humaniz Comput 8(4):625–635. https://doi.org/10.1007/s12652-017-0475-7

    Article  Google Scholar 

  26. Rautaray SS, Agrawal A (2015) Vision based hand gesture recognition for human computer interaction: a survey. Artif Intell Rev 43(1):1–54. https://doi.org/10.1007/s10462-012-9356-9

    Article  Google Scholar 

  27. Rittscher J, Blake A (1999) Classification of human body motion. In: Computer vision, 1999. The proceedings of the seventh IEEE international conference, vol 1, pp 634–639. https://doi.org/10.1109/ICCV.1999.791284

  28. Sahbi H, Geman D (2006) A hierarchy of support vector machines for pattern detection. J Mach Learn Res 7:2087–2123

    MathSciNet  MATH  Google Scholar 

  29. Shen X, Hua G, Williams L, Wu Y (2012) Dynamic hand gesture recognition: an exemplar-based approach from motion divergence fields. Image Vis Comput 30(3):227–235. https://doi.org/10.1016/j.imavis.2011.11.003

    Article  Google Scholar 

  30. Silva EJ, Zanchettin CA (2016) Voronoi diagram based classifier for multiclass imbalanced data sets. In: Intelligent systems (BRACIS), 5th Brazilian conference, pp 109–114. https://doi.org/10.1109/BRACIS.2016.030

  31. Singha J, Laskar RH (2016a) Self co-articulation detection and trajectory guided recognition for dynamic hand gestures. IET Comput Vision 10(2):143–152. https://doi.org/10.1049/iet-cvi.2014.0432

    Article  Google Scholar 

  32. Singha J, Laskar RH (2016b) Recognition of global hand gestures using self co-articulation information and classifier fusion. J Multimodal User Interfaces 10(1):77–93. https://doi.org/10.1007/s12193-016-0212-0

    Article  Google Scholar 

  33. Singha J, Laskar RH (2017) Hand gesture recognition using two-level speed normalization, feature selection and classifier fusion. Multimedia Syst 23(4):499–514. https://doi.org/10.1007/s00530-016-0510-0

    Article  Google Scholar 

  34. Singha J, Misra S, Laskar RH (2016) Effect of variation in gesticulation pattern in dynamic hand gesture recognition system. Neurocomputing 208:269–280. https://doi.org/10.1016/j.neucom.2016.05.049

    Article  Google Scholar 

  35. Singha J, Roy A, Laskar RH (2018) Dynamic hand gesture recognition using vision-based approach for human–computer interaction. Neural Comput Appl 29(4):1129–1141. https://doi.org/10.1007/s00521-016-2525-z

    Article  Google Scholar 

  36. Sun CT, Jang JS (1993) A neuro-fuzzy classifier and its applications. In: Fuzzy systems, second IEEE international conference, pp 94–98. https://doi.org/10.1109/FUZZY.1993.327457

  37. Thabet E, Khalid F, Sulaiman PS, Yaakob R (2018) Fast marching method and modified features fusion in enhanced dynamic hand gesture segmentation and detection method under complicated background. J Ambient Intell Humaniz Comput 9(3):755–769. https://doi.org/10.1007/s12652-017-0512-6

    Article  Google Scholar 

  38. Thiam P, Kessler V, Schwenker F (2017) Hierarchical combination of video features for personalised pain level recognition. In: 25th European symposium on artificial neural networks, computational intelligence and machine learning, pp 465–470

  39. Yamato J, Ohya J, Ishii K (1992) Recognizing human action in time-sequential images using hidden markov model. In: Computer vision and pattern recognition, IEEE computer society conference, pp 379–385. https://doi.org/10.1109/CVPR.1992.223161

  40. Yilmaz A, Shah M (2005) Actions sketch: a novel action representation. In: Computer vision and pattern recognition, IEEE computer society conference on, vol 1, pp 984–989. https://doi.org/10.1109/CVPR.2005.58

  41. Yoon HS, Soh J, Bae YJ, Yang HS (2001) Hand gesture recognition using combined features of location, angle and velocity. Pattern Recognit 34(7):1491–1501. https://doi.org/10.1016/S0031-3203(00)00096-0

    Article  MATH  Google Scholar 

  42. Yun SS, Nguyen Q, Choi J (2017) Recognition of emergency situations using audio–visual perception sensor network for ambient assistive living. J Ambient Intell Humaniz Comput. https://doi.org/10.1007/s12652-017-0597-y

    Article  Google Scholar 

  43. Zeng W, Wang C, Wang Q (2018) Hand gesture recognition using leap motion via deterministic learning. Multimedia Tools Appl. https://doi.org/10.1007/s11042-018-5998-1

    Article  Google Scholar 

  44. Zhang Z, Tian Z, Zhou M (2018) HandSense: smart multimodal hand gesture recognition based on deep neural networks. J Ambient Intell Humaniz Comput. https://doi.org/10.1007/s12652-018-0989-7

    Article  Google Scholar 

Download references

Acknowledgements

Authors are thankful to the Ministry of Electronics and Information Technology, Government of India for the financial support under the Visvesvaraya Ph.D. scheme. Authors also acknowledge Speech and Image Processing Laboratory, ECE Dept., National Institute of Technology Silchar, India, for providing all necessary facilities to carry out the research work.

Funding

The study was funded by Ministry of Electronics and Information Technology, Government of India with Grant no. PhD-MLA/4(74)/2015-16.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Songhita Misra.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Misra, S., Laskar, R.H. Development of a hierarchical dynamic keyboard character recognition system using trajectory features and scale-invariant holistic modeling of characters. J Ambient Intell Human Comput 10, 4901–4923 (2019). https://doi.org/10.1007/s12652-019-01189-2

Download citation

Keywords

  • Holistic model
  • Dynamic gestures
  • Keyboard characters
  • Hierarchical model
  • Trajectory features