Skip to main content
SpringerLink
Log in

Gesture Interaction in Virtual Reality

A Low-Cost Machine Learning System and a Qualitative Assessment of Effectiveness of Selected Gestures vs. Gaze and Controller Interaction

  • Conference paper
  • First Online:
Human-Computer Interaction – INTERACT 2021 (INTERACT 2021)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 12934))

Included in the following conference series:

Abstract

We explore gestures as interaction methods in virtual reality (VR). We detect hand and body gestures using human pose estimation based on off-the-shelf optical camera images using machine learning, and obtain reliable gesture recognition without additional sensors. We then employ an avatar to prompt users to learn and use gestures to communicate. Finally, to understand how well gestures serve as interaction methods, we compare the studied gesture-based interaction methods with baseline common interaction modalities in VR (controllers, gaze interaction) in a pilot study including usability testing.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Weissmann, J., Salomon, R.: Gesture recognition for virtual reality applications using data gloves and neural networks. In: Proceedings of the International Joint Conference on Neural Networks, vol. 3, pp. 2043–2046. IEEE (1999)

    Google Scholar 

  2. Lin, W., Du, L., Harris-Adamson, C., Barr, A., Rempel, D.: Design of hand gestures for manipulating objects in virtual reality. In: Kurosu, Masaaki (ed.) HCI 2017. LNCS, vol. 10271, pp. 584–592. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-58071-5_44

    Chapter  Google Scholar 

  3. Frey, G., Jurkschat, A., Korkut, S., Lutz, J., Dornberger, R.: Iintuitive hand gestures for the interaction with information visualizations in virtual reality. In: Tom Dieck, M.C., Jung, T. (eds.) Augmented Reality and Virtual Reality: The Power of AR and VR for Business, pp. 261–273. Springer International Publishing, Cham (2019)

    Chapter  Google Scholar 

  4. Cabral, M.C., Morimoto, C.H., Zuffo, M.K.: On the usability of gesture interfaces in virtual reality environments. In: Proceedings of the 2005 Latin American Conference on Human-Computer Interaction - CLIHC 2005, pp. 100–108. ACM Press, New York (2005)

    Google Scholar 

  5. Çöltekin, A., et al.: Geospatial information visualization and extended reality displays. In: Guo, Huadong, Goodchild, Michael F., Annoni, Alessandro (eds.) Manual of Digital Earth, pp. 229–277. Springer, Singapore (2020). https://doi.org/10.1007/978-981-32-9915-3_7

    Chapter  Google Scholar 

  6. Çöltekin, A., et al.: Extended reality in spatial sciences: a review of research challenges and future directions. ISPRS Int. J. Geo Inf. 9(7), 439 (2020)

    Article  Google Scholar 

  7. Çöltekin, A., Hempel, J., Brychtova, A., Giannopoulos, I., Stellmach, S., Dachselt, R.: Gaze and feet as additional input modalities for interaction with geospatial interfaces. ISPRS Ann. Photogramm. Remote Sens. Spat. Inf. Sci. III-2, 113–120 (2016)

    Google Scholar 

  8. Maloney, D., Freeman, G., Wohn, D.Y.: Talking without a voice. In: Proceedings of the ACM Human-Computer Interaction, vol. 4, pp. 1–25 (2020)

    Google Scholar 

  9. Valliappan, N., et al.: Accelerating eye movement research via accurate and affordable smartphone eye tracking. Nat. Commun. 11, 4553 (2020)

    Article  Google Scholar 

  10. Morris, D.: Bodytalk: A World Guide to Gestures. Jonathan Cape, London (1994)

    Google Scholar 

  11. Pease, A.: Body Language: How to Read Others’ Thoughts by Their Gestures. Sheldon Press, London (1988)

    Google Scholar 

  12. Brooke, J.: SUS - a quick and dirty usability scale usability and context. In: Jordan, P.W., Thomas, B., McClelland, I.L., Weerdmeester, B. (eds.) Usability Evaluation in Industry, pp. 189–196 (1996)

    Google Scholar 

  13. Nielsen, J.: How many test users in a usability study? https://www.nngroup.com/articles/how-many-test-users/. Accessed 10 June 2021

  14. Oved, D.: Real-time human pose estimation in the browser with TensorFlow.js | by TensorFlow | TensorFlow | Medium. https://medium.com/tensorflow/real-time-human-pose-estimation-in-the-browser-with-tensorflow-js-7dd0bc881cd5. Accessed 10 June 2021

  15. Cao, Z., Hidalgo, G., Simon, T., Wei, S.-E., Sheikh, Y.: OpenPose: realtime multi-person 2D pose estimation using part affinity fields. IEEE Trans. Pattern Anal. Mach. Intell. 43, 172–186 (2018)

    Article  Google Scholar 

  16. Bazarevsky, V., Grishchenko, I., Raveendran, K., Zhu, T., Zhang, F., Grundmann, M.: BlazePose: on-device real-time body pose tracking. arXiv (2020)

    Google Scholar 

  17. Abadi, M., et al.: TensorFlow: a system for large-scale machine learning. In: Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2016 (2016)

    Google Scholar 

  18. Shanmugamani, R.: Deep learning for computer vision: expert techniques to train advanced neural networks using TensorFlow and Keras. Birmingham Mumbai: Packt (2018)

    Google Scholar 

  19. Ahlawat, S., Batra, V., Banerjee, S., Saha, J., Garg, A.: Hand gesture recognition using convolutional neural network. In: Bhattacharyya, S., Hassanien, A.E., Gupta, D., Khanna, A., Pan, I. (eds.) International Conference on Innovative Computing and Communications. LNNS, vol. 56, pp. 179–186. Springer, Singapore (2019). https://doi.org/10.1007/978-981-13-2354-6_20

    Chapter  Google Scholar 

  20. Hayashi, E., et al.: RadarNet: efficient gesture recognition technique utilizing a miniature radar sensor. In: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1–14 (2021)

    Google Scholar 

  21. Golash, R., Jain, Y.K.: Trajectory-based cognitive recognition of dynamic hand gestures from webcam videos. Int. J. Eng. Res. Technol. 13(6), 1432–1440 (2020). ISSN 0974-3154

    Google Scholar 

  22. Agrawal, M., Ainapure, R., Agrawal, S., Bhosale, S., Desai, S.: Models for hand gesture recognition using deep learning. In: 2020 IEEE 5th International Conference on Computing Communication and Automation (ICCCA), pp. 589–594. IEEE (2020)

    Google Scholar 

  23. Bendarkar, D., Somase, P., Rebari, P., Paturkar, R., Khan, A.: Web based recognition and translation of American sign language with CNN and RNN. International Association of Online Engineering (2021). https://www.learntechlib.org/p/218958/

  24. Khan, O., Ahmed, I., Cottingham, J., Rahhal, M., Arvanitis, T.N., Elliott, M.T.: Timing and correction of stepping movements with a virtual reality avatar. PlosOne 15(2), e0229641 (2020)

    Google Scholar 

  25. Chan, C., Ginosar, S., Zhou, T., Efros, A.: Everybody dance now. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Korea (South), October 2019, pp. 5932–5941 (2019)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Institute of Interactive Technologies, University of Applied Sciences and Arts Northwestern Switzerland, Brugg-Windisch, Switzerland

    Cloe Huesser & Arzu Çöltekin

  2. Esri Inc., Zurich R&D, Zurich, Switzerland

    Simon Schubiger

Authors
  1. Cloe Huesser
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Simon Schubiger
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Arzu Çöltekin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Arzu Çöltekin .

Editor information

Editors and Affiliations

  1. Department of Electrical and Information Engineering, Polytechnic University of Bari, Bari, Italy

    Carmelo Ardito

  2. Computer Science Department, University of Bari Aldo Moro, Bari, Italy

    Rosa Lanzilotti

  3. Computer Science Department, University of Pisa, Pisa, Italy

    Alessio Malizia

  4. Department of Computer Science, University of York, York, UK

    Helen Petrie

  5. Computer Science Department, University of Bari Aldo Moro, Bari, Italy

    Antonio Piccinno

  6. Computer Science Department, University of Bari Aldo Moro, Bari, Italy

    Giuseppe Desolda

  7. Microsoft Research, Redmond, WA, USA

    Kori Inkpen

Rights and permissions

Reprints and permissions

Copyright information

© 2021 IFIP International Federation for Information Processing

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Huesser, C., Schubiger, S., Çöltekin, A. (2021). Gesture Interaction in Virtual Reality. In: Ardito, C., et al. Human-Computer Interaction – INTERACT 2021. INTERACT 2021. Lecture Notes in Computer Science(), vol 12934. Springer, Cham. https://doi.org/10.1007/978-3-030-85613-7_11

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-85613-7_11

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-85612-0

  • Online ISBN: 978-3-030-85613-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation