Mirrored Motion: Augmenting Reality and Implementing Whole Body Gestural Control Using Pervasive Body Motion Capture Based on Wireless Sensors

  • Philip Smit
  • Peter Barrie
  • Andreas Komninos
  • Oleksii Mandrychenko
Part of the Human-Computer Interaction Series book series (HCIS)


There has been a lot of discussion in recent years around the ­disappearing computer concept and most of the results of that discussion have been realized in the form of mobile devices and applications. What has got lost a little in this discussion is the moves that have seen the miniaturization of sensors that can be wirelessly attached to places and to humans in order to provide a new type of free flowing interaction. In order to investigate what these new sensors could achieve and at what cost, we implemented a configurable, wearable motion-capture system based on wireless sensor nodes, requiring no special environment to operate in. We discuss the system architecture and discuss the implications and opportunities afforded by it for innovative HCI design. As a practical application of the technology, we describe a prototype implementation of a pervasive, wearable augmented reality (AR) system based on the motion-capture system. The AR application uses body motion to visualize and interact with virtual objects populating AR settings. Body motion is used to implement a whole body gesture-driven interface to manipulate the virtual objects. Gestures are mapped to corresponding behaviours for virtual objects, such as controlling the playback and volume of virtual audio players or displaying a virtual object’s metadata.


Sensor Node Augmented Reality Virtual World Gesture Recognition Virtual Object 


  1. 1.
    AMELIA: A generic library for pattern recognition and generation. http://ame4.hc.asu.edu/amelia/(2010) (link valid 08/2010)
  2. 2.
    Antifakos, S., Schiele, B.: Bridging the gap between virtual and physical games using wearable sensors. In: Proceedings of the Sixth International Symposium on Wearable Computers (ISWC 2002), pp. 139–140, Seattle (2002)Google Scholar
  3. 3.
    Azuma, R.: A survey of augmented reality. Presence Teleoper. Virtual Environ. 6(4), 355–385 (August 1997)Google Scholar
  4. 4.
    Bachmann, E.: Inertial and magnetic angle tracking of limb segments for inserting humans into synthetic environments. Ph.D. thesis, Naval Postgraduate School (2000)Google Scholar
  5. 5.
    Bodenheimer, B., Rose, C., Pella, J., Rosenthal, S.: The process of motion capture: dealing with the data. In: Computer Animation and Simulation, pp.    3–18, Milano. Eurographics, Springer, London (1997)Google Scholar
  6. 6.
    Bonato, P.: Wearable sensors/systems and their impact on biomedical engineering. Eng. Med. Biol. Mag. IEEE 22(3), 18–20 (2003)CrossRefGoogle Scholar
  7. 7.
    Buchmann, V., Violich, S., Billinghurst, M., Cockburn, A.: FingARtips: gesture based direct manipulation in Augmented Reality. In: GRAPHITE 2004: Proceedings of the 2nd International Conference on Computer Graphics and Interactive Techniques in Australasia and South East Asia, Singapore (2004)Google Scholar
  8. 8.
    Clarkson, B.P.: Life patterns: structure from wearable sensors. Ph.D. thesis, Massachusetts Institute of Technology (2002)Google Scholar
  9. 9.
    Crossan, A., Williamson, J., Brewster, S., Murray-Smith, R.: Wrist rotation for interaction in mobile contexts. In: Proceedings of the 10th International Conference on Human Computer Interaction with Mobile Devices and Services, pp. 435–438. ACM, Amsterdam (2008)Google Scholar
  10. 10.
    Jovanov, E.: Wireless technology and system integration in body area networks for m-Health applications. In: Proceedings of the 27th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Shanghai (2005)Google Scholar
  11. 11.
    Lyu, M.R., King, I., Wong, T.T., Yau, E., Chan, P.W.: ARCADE: augmented reality computing arena for digital entertainment. In: 5th IEEE Aerospace Conference, Big Sky (2005)Google Scholar
  12. 12.
    Martins, T., Sommerer, C., Mignonneau, L., Correia, N.: Gauntlet: a wearable interface for ubiquitous gaming. In: Proceedings of the 10th International Conference on Human Computer Interaction with Mobile Devices and Services, pp. 367–370, Amsterdam. ACM, New York (2008)Google Scholar
  13. 13.
    Molet, T., Boulic, R., Thalmann, D.: Human motion capture driven by orientation measurements. Presence Teleoper. Virtual Environ. 8(2), 187–203 (1999)CrossRefGoogle Scholar
  14. 14.
    O’Brien, J.F., Bodenheimer, R.E., Brostow, G.J., Hodgins, J.K.: Automatic joint parameter estimation from magnetic motion capture data. In: Proceedings of Graphics Interface 2000, pp. 53–60, Montréal (2000)Google Scholar
  15. 15.
    Ouchi, K., Suzuki, T., Doi, M.: LifeMinder: a wearable healthcare support system using user’s context. In: Proceedings of the 22nd International Conference on Distributed Computing Systems, pp. 791–792, Vienna (2002)Google Scholar
  16. 16.
    Rajko, S., Oian, G.: HMM parameter reduction for practical gesture recognition. In: IEEE International Conference on Face and Gesture Recognition, Amsterdam (2008)Google Scholar
  17. 17.
    Rajko, S., Oian, G., Ingalls, T., James, J.: Real-time gesture recognition with minimal training requirements and on-line learning. IEEE Conference on Computer Vision and Pattern Recognition, Minneapolis (2007)Google Scholar
  18. 18.
    Seon-Woo, L., Mase, K.: Activity and location recognition using wearable sensors. Pervasive Comput. IEEE 1(3), 24–32 (2002)CrossRefGoogle Scholar
  19. 19.
    Svensson, A., Björk, S., Åkesson, K.P.: Tangible handimation: real-time animation with a sequencer-based tangible interface. In: Proceedings of the 5th Nordic Conference on Human Computer Interaction, Lund (2008)Google Scholar
  20. 20.
    Tognetti, A., Lorussi, F., Tesconi, M., Bartalesi, R., Zupone, G., De Rossi, D.: Wearable kinesthetic systems for capturing and classifying body posture and gesture. Conf. Proc. IEEE Eng. Med. Biol. Soc. 1, 1012–1015 (2005)Google Scholar
  21. 21.
    Vlasic, D., Adelsberger, R., Vanucci, G., Barnwell, J., Gross, M., Matusik, W., Popovic, J.: Practical motion capture in everyday surroundings. ACM Trans. Graph. 26(3), 35 (2007)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag London Limited 2011

Authors and Affiliations

  • Philip Smit
    • 1
  • Peter Barrie
    • 1
  • Andreas Komninos
    • 1
  • Oleksii Mandrychenko
    • 1
  1. 1.School of Engineering and ComputingGlasgow Caledonian UniversityGlasgowUK

Personalised recommendations