A New Virtual Keyboard with Finger Gesture Recognition for AR/VR Devices
Abstract
This paper proposes a system that types the virtual keyboard by recognizing hand gestures in a single camera based environment. A virtual keyboard is designed in a multi-tab method of 3 × 4 arrays that are widely used in a mobile environment. In order to make it easy to identify the type-in action and consequently, to increase the type-in speed, this paper proposes a new definition of type-in action as the contact between the thumb and the index finger. To further reduce the input time, the keyboard layout is optimized by efficiently arranging alphabet keys according to the frequency of character appearance. Experimental results show that the proposed type-in action is effective to give commands with a virtual keyboard. Furthermore, the proposed keyboard layout achieves the speed-up by an average of 46.16% compared to the most conventional ‘ABC’ keyboard.
Keywords
Hand gesture Virtual keyboard Convex-hull Keyboard layout optimization Hand recognitionNotes
Acknowledgments
This work was supported by the World Class 300 Project (R&D) (S2482780, Development of Core Technologies for 3D Sensing Camera Module) of the SMBA(Korea) and by “The Project of Industrial Technology Innovation” through the Ministry of Trade, Industry and Energy(MOTIE) (10082585,2017).
References
- 1.Kato, H., Billinghurst, M., Poupyrev, I., Imamoto, K., Tachibana, K.: Virtual object manipulation on a table-top AR environment. In: IEEE and ACM International Symposium on Augmented Reality, pp. 111–119, October 2000Google Scholar
- 2.Lee, T., Hollerer, T.: Handy AR: Markerless inspection of augmented reality objects using fingertip tracking. In: 11th IEEE International Symposium on Wearable Computers, pp. 83–90, October 2007Google Scholar
- 3.LaViola, J.J.: A survey of hand posture and gesture recognition techniques and technology. Technical report CS-99-11. Brown University, Providence (1999)Google Scholar
- 4.Argyros, A.A., Lourakis, M.I.A.: Vision-based interpretation of hand gestures for remote control of a computer mouse. In: Huang, T.S., Sebe, N., Lew, M.S., Pavlović, V., Kölsch, M., Galata, A., Kisačanin, B. (eds.) ECCV 2006. LNCS, vol. 3979, pp. 40–51. Springer, Heidelberg (2006). https://doi.org/10.1007/11754336_5CrossRefGoogle Scholar
- 5.Liu, N., Lovell, B.C.: Hand gesture extraction by active shape models. In: Proceedings of the Digital Imaging Computing: Techniques and Applications, pp. 1–6, December 2005Google Scholar
- 6.Lockton, R., Fitzgibbon, A.W.: Real-time gesture recognition using deterministic boosting. In: Proceedings of British Machine Vision Conference, vol. 2, pp. 817–826, September 2002Google Scholar
- 7.Lee, J.W., Lee, K.S., Lee, H.J.: 3D-convolutional neural network for efficient hand gesture recognition. In: International Conference on Electronics, Information, and Communication, January 2017Google Scholar
- 8.Lee, H.S., Lee, D.H., Kim, J.S., Lee, H.J.: Fast hand gesture recognition with CNN and feature matching. In: 30th Workshop on Image Processing and Image Understanding, February 2018Google Scholar
- 9.Kolsch, M., Turk, M.: Keyboards without keyboards: a survey of virtual keyboards. Technical report 2002-21. University of California, Santa Barbara, July 2002Google Scholar
- 10.Sarkar, A.R., Sanyal, G., Majumder, S.: Hand gesture recognition systems: a survey. Int. J. Comput. Appl. 71(15), 0975–8887 (2013)Google Scholar
- 11.Qin, S., Zhu, X., Yang, Y.: Real-time hand gesture recognition from depth images using convex shape decomposition method. J. Sig. Process. 74(1), 47–58 (2014)CrossRefGoogle Scholar
- 12.Markussen, A., Jakobsen, M.R., Hornbæk, K.: Vulture: a mid-air word-gesture keyboard. In: Proceedings of CHI, pp. 1073–1082. ACM (2014)Google Scholar
- 13.Bergounioux, M.: Applications. Introduction au traitement mathématique des Images - méthodes déterministes. MA, vol. 76, pp. 157–176. Springer, Heidelberg (2015). https://doi.org/10.1007/978-3-662-46539-4_7CrossRefzbMATHGoogle Scholar
- 14.MacKenzie, I.S., Zhang, S.X.: The design and evaluation of a high-performance soft keyboard. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 25–31, May 1999Google Scholar
- 15.Levine, S.H., Trepagnier, G.C., Getschow, C.O., Minneman, S.L.: Multi-character key text entry using computer disambiguation. In: Proceedings of the Tenth Annual Conference on Rehabilitation Engineering, pp. 177–179, June 1987Google Scholar
- 16.Foulds, R.A., Soede, M., Van Balkom, H.: Statistical disambiguation of multi-character keys applied to reduce motor requirements for augmentative and alternative communication. Altern. Augment. Commun. 3, 192–195 (1987)CrossRefGoogle Scholar
- 17.Bi, X., Zhai, S.: IJQwerty: what difference does one key change make? Gesture typing keyboard optimization bounded by one key position change from Qwerty. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, p. 49-8, May 2016Google Scholar
- 18.Arnott, J.L., Javed, M.Y.: Probabilistic character disambiguation for reduced keyboards using small text samples. Augment. Altern. Commun. 8(3), 215–223 (1992)CrossRefGoogle Scholar
- 19.Yin, P.Y., Su, E.P.: Cyber swarm optimization for general keyboard arrangement problem. Int. J. Ind. Ergonomics 41, 43–52 (2011)CrossRefGoogle Scholar
- 20.Fitts, P.M.: The information capacity of the human motor system in controlling the amplitude of movement. J. Exp. Psychol. 47, 381–391 (1954)CrossRefGoogle Scholar
- 21.Smith, B.A., Bi, X., Zhai, S.: Optimizing touchscreen keyboards for gesture typing. In: Proceedings of the CHI 2015 Conference on Human Factors in Computer Systems, pp. 18–23, April 2015Google Scholar
- 22.Zhai, S., Hunter, M., Smith, B.A.: The metropolis keyboard: an exploration of quantitative techniques for virtual keyboard design. In: Proceedings of the UIST 2000 Symposium on User Interface Software and Technology, pp. 119–128, November 2000Google Scholar
- 23.Kovac, J., Peer, P., Solina, F.: Human skin color clustering for face detection. In: International Conference on Computer as a Tool, EUROCON 2003, pp. 144–148, September 2003Google Scholar
- 24.Lee, T.H.: A new keyboard typing system with hand gesture recognition. Doctoral dissertation, Seoul National University, February 2018Google Scholar