Signal-Processing Transformation from Smartwatch to Arm Movement Gestures

Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 781)


This paper concerns virtual reality (VR) environments and innovative, natural interaction techniques for them. The presented research was driven by the goal to enable users to invoke actions with their body physically, causing the correct action of the VR environment. The paper introduces a system that tracks a user’s movements that are recognized as specific gestures. Smartwatches are promising new devices enabling new modes of interaction. They can support natural, hands-free interaction. The presented effort is concerned with the replacement of common touch input gestures with body movement gestures. Missing or insufficiently precise sensor data are a challenge, e.g., gyroscope and magnetometer data. This data is needed, together with acceleration data, to compute orientation and motion of the device. A transformation of recorded smartwatch data to arm movement gestures is introduced, involving data smoothing and gesture state machines.


Intuitive and natural interaction Low budget interaction devices Mobile devices Virtual reality Body movement gestures Gesture recognition 



This research was funded by the German research foundation (DFG) within the IRTG 2057 “Physical Modeling for Virtual Manufacturing Systems and Processes”.


  1. 1.
    Rupprecht, F., Hamann, B., Weidig, C., Aurich, J., Ebert, A.: IN2CO - a visualization framework for intuitive collaboration. In: EuroVis - Short Papers. ACM (2016)Google Scholar
  2. 2.
    Pausch, R., Proffitt, D., Williams, G.: Quantifying immersion in virtual reality. In: Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques, pp. 13–18. ACM (1997)Google Scholar
  3. 3.
    Bryson, S., Feiner, S., Brooks Jr., F., Hubbard, P., Pausch, R., van Dam, A.: Research frontiers in virtual reality. In: Proceedings of the 21st Annual Conference on Computer (1994)Google Scholar
  4. 4.
    König, W.A., Rädle, R., Reiterer, H.: Squidy: a zoomable design environment for natural user interfaces. ACM (2009)Google Scholar
  5. 5.
    Ball, R., North, C., Bowman, D.: Move to improve: promoting physical navigation to increase user performance with large displays. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 191–200. ACM (2007)Google Scholar
  6. 6.
    Rupprecht, F., Ebert, A., Schneider, A., Hamann, B.: Virtual reality meets smartwatch: intuitive, natural, and multi-modal interaction. In: Proceedings of the 2017 Chi Conference Extended Abstracts on Human Factors in Computing Systems, pp. 2884–2890. ACM (2017)Google Scholar
  7. 7.
    Bergé, L.-P., Serrano, M., Perelman, G., Dubois, E.: Exploring smartphone-based interaction with overview + detail interfaces on 3D public displays. In: Proceedings of the 16th International Conference on Human-Computer Interaction with Mobile Devices & Services, pp. 125–134 (2014)Google Scholar
  8. 8.
    Tregillus, S., Folmer, E.: VR-STEP: walking-inplace using inertial sensing for hands free navigation in mobile VR environments. In: Proceedings of the 2016 Chi Conference on Human Factors in Computing Systems, pp. 1250–1255. ACM (2016)Google Scholar
  9. 9.
    Sharples, S., Cobb, S., Moody, A., Wilson, J.R.: Virtual reality induced symptoms and effects (VRISE): comparison of head mounted display (HMD), desktop and projection display systems. Displays 29(2), 58–69 (2008)CrossRefGoogle Scholar
  10. 10.
    Freeman, E., Brewster, S., Lantz, V.: Do that, there: an interaction technique for addressing in-air gesture systems. In: Proceedings of the 34th Annual Conference on Human Factors in Computing Systems, CHI 2016. ACM (2016)Google Scholar
  11. 11.
    Wilkinson, G., Kharrufa, A., Hook, J., Pursgrove, B., Wood, et al.: Expressy: using a wrist-worn inertial measurement unit to add expressiveness to touch-based interactions. In: Proceedings of the Conference on Human Factors in Computing Systems. ACM (2016)Google Scholar
  12. 12.
    Arefin Shimon, S.S., Lutton, C., Xu, Z., Morrison-Smith, S., Boucher, C., Ruiz, J.: Exploring nontouchscreen gestures for smartwatches. In: Proceedings of the 2016 Chi Conference on Human Factors in Computing Systems, pp. 3822–3833. ACM (2016)Google Scholar
  13. 13.
    Houben, S., Marquardt, N.: WatchConnect: a toolkit for prototyping smartwatch-centric cross-device applications. In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, pp. 1247–1256. ACM (2015)Google Scholar
  14. 14.
    Mace, D., Gao, W., Coskun, A.K.: Improving accuracy and practicality of accelerometer-based hand gesture recognition. In: Interacting with Smart Objects, vol. 45, pp. 45–49 (2013)Google Scholar
  15. 15.
    Mäntyjärvi, J., Kela, J., Korpipää, P., Kallio, S.: Enabling fast and effortless customisation in accelerometer based gesture interaction. In: Proceedings of the 3rd International Conference on Mobile and Ubiquitous Multimedia, pp. 25–31. ACM (2004)Google Scholar
  16. 16.
    Schlömer, T., Poppinga, B., Henze, N., Boll, S.: Gesture recognition with a Wii controller. In: Proceedings of the 2nd International Conference on Tangible and Embedded Interaction. ACM (2008)Google Scholar
  17. 17.
    Chu, H., Huang, S., Liaw, J.: An acceleration feature-based gesture recognition system. In: International Conference on Systems, Man, and Cybernetics, pp. 3807–3812. IEEE (2013)Google Scholar
  18. 18.
    Wagner, F., Schmuki, R., Wagner, T., Wolstenholme, P.: Modeling software with finite state machines: a practical approach. CRC Press (2006)Google Scholar
  19. 19.
    Wu, J., Pan, G., Zhang, D., Qi, G., Li, S.: Gesture recognition with a 3-D accelerometer. In: International Conference on Ubiquitous Intelligence and Computing. Springer (2009)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Computer Graphics and HCITechnische Universität KaiserslauternKaiserslauternGermany
  2. 2.Department of Computer ScienceUniversity of CaliforniaDavisUSA

Personalised recommendations