Advertisement

A New Virtual Keyboard with Finger Gesture Recognition for AR/VR Devices

  • Tae-Ho Lee
  • Hyuk-Jae Lee
Conference paper
First Online:
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10903)

Abstract

This paper proposes a system that types the virtual keyboard by recognizing hand gestures in a single camera based environment. A virtual keyboard is designed in a multi-tab method of 3 × 4 arrays that are widely used in a mobile environment. In order to make it easy to identify the type-in action and consequently, to increase the type-in speed, this paper proposes a new definition of type-in action as the contact between the thumb and the index finger. To further reduce the input time, the keyboard layout is optimized by efficiently arranging alphabet keys according to the frequency of character appearance. Experimental results show that the proposed type-in action is effective to give commands with a virtual keyboard. Furthermore, the proposed keyboard layout achieves the speed-up by an average of 46.16% compared to the most conventional ‘ABC’ keyboard.

Keywords

Hand gesture Virtual keyboard Convex-hull  Keyboard layout optimization Hand recognition 
This is a preview of subscription content, log in to check access.

Notes

Acknowledgments

This work was supported by the World Class 300 Project (R&D) (S2482780, Development of Core Technologies for 3D Sensing Camera Module) of the SMBA(Korea) and by “The Project of Industrial Technology Innovation” through the Ministry of Trade, Industry and Energy(MOTIE) (10082585,2017).

References

  1. 1.
    Kato, H., Billinghurst, M., Poupyrev, I., Imamoto, K., Tachibana, K.: Virtual object manipulation on a table-top AR environment. In: IEEE and ACM International Symposium on Augmented Reality, pp. 111–119, October 2000Google Scholar
  2. 2.
    Lee, T., Hollerer, T.: Handy AR: Markerless inspection of augmented reality objects using fingertip tracking. In: 11th IEEE International Symposium on Wearable Computers, pp. 83–90, October 2007Google Scholar
  3. 3.
    LaViola, J.J.: A survey of hand posture and gesture recognition techniques and technology. Technical report CS-99-11. Brown University, Providence (1999)Google Scholar
  4. 4.
    Argyros, A.A., Lourakis, M.I.A.: Vision-based interpretation of hand gestures for remote control of a computer mouse. In: Huang, T.S., Sebe, N., Lew, M.S., Pavlović, V., Kölsch, M., Galata, A., Kisačanin, B. (eds.) ECCV 2006. LNCS, vol. 3979, pp. 40–51. Springer, Heidelberg (2006).  https://doi.org/10.1007/11754336_5CrossRefGoogle Scholar
  5. 5.
    Liu, N., Lovell, B.C.: Hand gesture extraction by active shape models. In: Proceedings of the Digital Imaging Computing: Techniques and Applications, pp. 1–6, December 2005Google Scholar
  6. 6.
    Lockton, R., Fitzgibbon, A.W.: Real-time gesture recognition using deterministic boosting. In: Proceedings of British Machine Vision Conference, vol. 2, pp. 817–826, September 2002Google Scholar
  7. 7.
    Lee, J.W., Lee, K.S., Lee, H.J.: 3D-convolutional neural network for efficient hand gesture recognition. In: International Conference on Electronics, Information, and Communication, January 2017Google Scholar
  8. 8.
    Lee, H.S., Lee, D.H., Kim, J.S., Lee, H.J.: Fast hand gesture recognition with CNN and feature matching. In: 30th Workshop on Image Processing and Image Understanding, February 2018Google Scholar
  9. 9.
    Kolsch, M., Turk, M.: Keyboards without keyboards: a survey of virtual keyboards. Technical report 2002-21. University of California, Santa Barbara, July 2002Google Scholar
  10. 10.
    Sarkar, A.R., Sanyal, G., Majumder, S.: Hand gesture recognition systems: a survey. Int. J. Comput. Appl. 71(15), 0975–8887 (2013)Google Scholar
  11. 11.
    Qin, S., Zhu, X., Yang, Y.: Real-time hand gesture recognition from depth images using convex shape decomposition method. J. Sig. Process. 74(1), 47–58 (2014)CrossRefGoogle Scholar
  12. 12.
    Markussen, A., Jakobsen, M.R., Hornbæk, K.: Vulture: a mid-air word-gesture keyboard. In: Proceedings of CHI, pp. 1073–1082. ACM (2014)Google Scholar
  13. 13.
    Bergounioux, M.: Applications. Introduction au traitement mathématique des Images - méthodes déterministes. MA, vol. 76, pp. 157–176. Springer, Heidelberg (2015).  https://doi.org/10.1007/978-3-662-46539-4_7CrossRefzbMATHGoogle Scholar
  14. 14.
    MacKenzie, I.S., Zhang, S.X.: The design and evaluation of a high-performance soft keyboard. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 25–31, May 1999Google Scholar
  15. 15.
    Levine, S.H., Trepagnier, G.C., Getschow, C.O., Minneman, S.L.: Multi-character key text entry using computer disambiguation. In: Proceedings of the Tenth Annual Conference on Rehabilitation Engineering, pp. 177–179, June 1987Google Scholar
  16. 16.
    Foulds, R.A., Soede, M., Van Balkom, H.: Statistical disambiguation of multi-character keys applied to reduce motor requirements for augmentative and alternative communication. Altern. Augment. Commun. 3, 192–195 (1987)CrossRefGoogle Scholar
  17. 17.
    Bi, X., Zhai, S.: IJQwerty: what difference does one key change make? Gesture typing keyboard optimization bounded by one key position change from Qwerty. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, p. 49-8, May 2016Google Scholar
  18. 18.
    Arnott, J.L., Javed, M.Y.: Probabilistic character disambiguation for reduced keyboards using small text samples. Augment. Altern. Commun. 8(3), 215–223 (1992)CrossRefGoogle Scholar
  19. 19.
    Yin, P.Y., Su, E.P.: Cyber swarm optimization for general keyboard arrangement problem. Int. J. Ind. Ergonomics 41, 43–52 (2011)CrossRefGoogle Scholar
  20. 20.
    Fitts, P.M.: The information capacity of the human motor system in controlling the amplitude of movement. J. Exp. Psychol. 47, 381–391 (1954)CrossRefGoogle Scholar
  21. 21.
    Smith, B.A., Bi, X., Zhai, S.: Optimizing touchscreen keyboards for gesture typing. In: Proceedings of the CHI 2015 Conference on Human Factors in Computer Systems, pp. 18–23, April 2015Google Scholar
  22. 22.
    Zhai, S., Hunter, M., Smith, B.A.: The metropolis keyboard: an exploration of quantitative techniques for virtual keyboard design. In: Proceedings of the UIST 2000 Symposium on User Interface Software and Technology, pp. 119–128, November 2000Google Scholar
  23. 23.
    Kovac, J., Peer, P., Solina, F.: Human skin color clustering for face detection. In: International Conference on Computer as a Tool, EUROCON 2003, pp. 144–148, September 2003Google Scholar
  24. 24.
    Lee, T.H.: A new keyboard typing system with hand gesture recognition. Doctoral dissertation, Seoul National University, February 2018Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of Electrical Engineering, Inter-university Semiconductor Research CenterSeoul National UniversitySeoulKorea

Personalised recommendations

Cite paper

Buy options