Activity Recognition Based on Intra and Extra Manipulation of Everyday Objects

  • Dipak Surie
  • Fabien Lagriffoul
  • Thomas Pederson
  • Daniel Sjölie
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4836)


Recognizing activities based on an actor’s interaction with everyday objects is an important research approach within ubiquitous computing. We present a recognition approach which complement objects grabbed or released information with the object’s internal state changes (as an effect of intra manipulation) and the object’s external state changes with reference to other objects (as an effect of extra manipulation). The concept of Intra manipulation is inspired by the fact that many everyday objects change their internal state when manipulated by the human actor, while extra manipulation is motivated by the fact that humans commonly rearrange the spatial relations between everyday objects as part of their activities. A detailed evaluation of our prototype activity recognition system in virtual reality (VR) environment is presented as a “proof of concept”. We have obtained a recognition precision of 92% on the activity-level and 81% on the action-level among 15 everyday home activities. Virtual reality was used as a test-bed in order to speed up the design process of our activity recognition system, allowing us to compensate for the limitations with currently available sensing technologies and to compare the contributions of intra manipulation and extra manipulation for activity recognition.


Activity Recognition Context Awareness Ubiquitous Computing Wearable Computing Virtual Reality 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Fishkin, K., Consolvo, S., Rode, J., Ross, B., Smith, I., Souter, K.: Ubiquitous Computing Support for Skills Assessment in Medical School. In: UbiHealth Workshop at the Sixth International Conference on Ubiquitous Computing (2004)Google Scholar
  2. 2.
    Backman, A., Bodin, K., Bucht, G., Janlert, L.-E., Maxhall, M., Pederson, T., Sjölie, D., Sondell, B., Surie, D.: easyADL - Wearable Support System for Independent Life despite Dementia. In: CHI 2006. Workshop on Designing Technology for People with Cognitive Impairments, pp. 22–23 (April 2006)Google Scholar
  3. 3.
    Philipose, M., Fishkin, K., Perkowitz, M., Patterson, D., Fox, D., Kautz, H., Hähnel, D.: Inferring Activities from Interactions with Objects. IEEE Pervasive Computing, 50–57 (October 2004)Google Scholar
  4. 4.
    Patterson, D., Fox, D., Kautz, H., Philipose, M.: Fine-Grained Activity Recognition by Aggregating Abstract Object Usage. In: Ninth IEEE International Symposium on Wearable Computers (2005)Google Scholar
  5. 5.
    Nardi, B. (ed.): Context and Consciousness: Activity Theory and Human-Computer Interaction. MIT Press, Cambridge (1995)Google Scholar
  6. 6.
    Surie, D., Pederson, T., Lagriffoul, F., Janlert, L.-E., Sjölie, D.: Activity Recognition using an Egocentric Perspective of Everyday Objects. In: UIC 2007. Proceedings of IFIP 2007 International Conference on Ubiquitous Intelligence and Computing, LNCS, vol. 4611, pp. 246–257. Springer, Heidelberg (July 2007)Google Scholar
  7. 7.
    Ward, J.A., Lukowicz, P., Troster, G., Starner, T.: Activity Recognition of Assembly Tasks Using Body-Worn Microphones and Accelerometers. Pattern Analysis and Machine Intelligence, IEEE Transactions 28(10), 1553–1567 (2006)CrossRefGoogle Scholar
  8. 8.
    Bao, L., Intille, S.: Activity Recognition from User-Annotated Acceleration Data. In: Ferscha, A., Mattern, F. (eds.) PERVASIVE 2004. LNCS, vol. 3001, pp. 1–17. Springer, Heidelberg (2004)Google Scholar
  9. 9.
    Lee, S.W., Mase, K.: Activity and location recognition using wearable sensors. IEEE Pervasive Computing 1(3), 24–32 (2002)CrossRefGoogle Scholar
  10. 10.
    Mayol, W., Murray, D.: Wearable hand activity recognition for event summarization. In: Ninth IEEE International Symposium on Wearable Computers, pp. 122–129 (October 2005)Google Scholar
  11. 11.
    Tapia, E., Intille, S., Larson, K.: Activity Recognition in the Home Using Simple and Ubiquitous Sensors. In: Ferscha, A., Mattern, F. (eds.) PERVASIVE 2004. LNCS, vol. 3001, pp. 158–175. Springer, Heidelberg (2004)Google Scholar
  12. 12.
    Hutchins, E.: Cognition in the Wild. MIT Press, Cambridge (1995)Google Scholar
  13. 13.
    Pederson, T.: From Conceptual Links to Causal Relations Physical-Virtual Artefacts in Mixed-Reality Space. PhD thesis, Dept. of Computing Science, Umeå university, report UMINF-03.14 (2003) ISBN 91-7305-556-5Google Scholar
  14. 14.
    Backman, A.: Colosseum3D - Authoring Framework for Virtual Environments. In: Proceedings of EUROGRAPHICS Workshop IPT & EGVE Workshop, pp. 225–226 (2005)Google Scholar
  15. 15.
    Rabiner, L.: A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition. In: Proceedings of the IEEE, vol. 77(2), IEEE Computer Society Press, Los Alamitos (1989)Google Scholar
  16. 16.
    Chen, J., Kam, A., Zhang, J., Liu, N., Shue, L.: Bathroom Activity Monitoring Based on Sound. In: Gellersen, H.-W., Want, R., Schmidt, A. (eds.) PERVASIVE 2005. LNCS, vol. 3468, pp. 47–61. Springer, Heidelberg (2005)Google Scholar
  17. 17.
    Lukowicz, P., Ward, J., Junker, H., Stäger, M., Tröster, G., Atrash, A., Starner, T.: Recognizing Workshop Activity Using Body Worn Microphones and Accelerometers. In: Ferscha, A., Mattern, F. (eds.) PERVASIVE 2004. LNCS, vol. 3001, pp. 18–32. Springer, Heidelberg (2004)Google Scholar
  18. 18.
    Lester, J., Choudhury, T., Kern, N., Borriello, G., Hannaford, B.: A Hybrid Discriminative/Generative Approach for Modeling Human Activities. In: 19th International Joint Conference on Artificial Intelligence (2005)Google Scholar
  19. 19.
    AMPS.: (as on January 2, 2007),
  20. 20.
    Finkenzeller, K.: RFID Handbook, 2nd edn. John Wiley and Sons, New York (2003)Google Scholar
  21. 21.
    Pederson, T.: Magic Touch: A Simple Object Location Tracking System Enabling the Development of Physical-Virtual Artefacts in Office Environments. In: Journal of Personal and Ubiquitous Computing, vol. 5, pp. 54–57. Springer, Heidelberg (2001)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2007

Authors and Affiliations

  • Dipak Surie
    • 1
  • Fabien Lagriffoul
    • 1
  • Thomas Pederson
    • 1
  • Daniel Sjölie
    • 2
  1. 1.Department of Computing Science, Umeå University, S-901 87 UmeåSweden
  2. 2.VRlab / HPC2N, Umeå University, S-901 87 UmeåSweden

Personalised recommendations