Using Multi-modal Sensing for Human Activity Modeling in the Real World

  • Beverly L. Harrison
  • Sunny Consolvo
  • Tanzeem Choudhury


Traditionally smart environments have been understood to represent those (often physical) spaces where computation is embedded into the users’ surrounding infrastructure, buildings, homes, and workplaces. Users of this “smartness” move in and out of these spaces. Ambient intelligence assumes that users are automatically and seamlessly provided with context-aware, adaptive information, applications and even sensing – though this remains a significant challenge even when limited to these specialized, instrumented locales. Since not all environments are “smart” the experience is not a pervasive one; rather, users move between these intelligent islands of computationally enhanced space while we still aspire to achieve a more ideal anytime, anywhere experience. Two key technological trends are helping to bridge the gap between these smart environments and make the associated experience more persistent and pervasive. Smaller and more computationally sophisticated mobile devices allow sensing, communication, and services to be more directly and continuously experienced by user. Improved infrastructure and the availability of uninterrupted data streams, for instance location-based data, enable new services and applications to persist across environments.


Physical Activity Activity Recognition Ubiquitous Computing Step Count Bike Ride 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [1]
    Bao, L. & Intille, S.S., Activity Recognition from User-Annotated Acceleration Data, In Proceedings of Pervasive ’04, (2004), 1-17.Google Scholar
  2. [2]
    Choudhury, T., Borriello, G., Consolvo, S., Haehnel, D., Harrison, B., Hemingway, B., Hightower, J., Klasnja, P., Koscher, K., LaMarca, A., Landay, J.A., LeGrand, L., Lester, J., Rahimi, A., Rea, A., & Wyatt, D. (Apr-Jun 2008). “The Mobile Sensing Platform: An Embedded Activity Recognition System, IEEE Pervasive Computing Magazine Special Issue on Activity-Based Computing, 7(2), 32-41.Google Scholar
  3. [3]
    Consolvo, S., Everitt, K., Smith, I., & Landay, J.A. (2006). Design Requirements for Technologies that Encourage Physical Activity, Proceedings of the Conference on Human Factors and Computing ystems: CHI 2006, (Montreal, Canada), New York, NY, USA: ACM Press, 457-66.Google Scholar
  4. [4]
    Consolvo, S., Klasnja, P., McDonald, D., Avrahami, D., Froehlich, J., LeGrand, L., Libby, R., Mosher, K., & Landay, J.A. (Sept 2008). Flowers or a Robot Army? Encouraging Awareness & Activity with Personal, Mobile Displays, Proceedings of the 10 th International Conference on Ubiquitous Computing: UbiComp 2008, (Seoul, Korea), New York, NY, USA: ACM PressGoogle Scholar
  5. [5]
    Consolvo, S., McDonald, D.W., Toscos, T., Chen, M.Y., Froehlich, J., Harrison, B., Klasnja, P., LaMarca, A., LeGrand, L., Libby, R., Smith, I., & Landay, J.A. (Apr 2008). Activity Sensing in the Wild: A Field Trial of UbiFit Garden, Proceedings of the Conference on Human Factors and Computing Systems: CHI 2008, (Florence, Italy), New York, NY, USA: ACM Press, 1797-806.Google Scholar
  6. [6]
    Fishkin, K. P., Jiang, B., Philipose, M. and Roy, S. I Sense a Disturbance in the Force: Long-range etection of Interactions with RFID-tagged Objects. Proceedings of the 6 th International Conference on Ubiquitous Computing: Ubicomp 2004, pp. 268-282.Google Scholar
  7. [7]
    Fogg, B.J. (2003). Persuasive Technology: Using Computers to Change What We Think and Do. San Francisco, CA, USA: Morgan Kaufmann Publishers.Google Scholar
  8. [8]
    Hightower, J., Consolvo, S., LaMarca, A., Smith, I. E., Hughes, J. Learning and Recognizing the Places We Go. Proceedings of the 7 th International Conference on Ubiquitous Computing: Ubicomp 2005: 159-176.Google Scholar
  9. [9]
    Lester, J. Choudhury, T., & Borriello, G. (2006). A Practical Approach to Recognizing Physical Activities, In Proceedings of the 4 th International Conference on Pervasive Computing: Pervasive ’06, Dublin, Ireland, 1-16.Google Scholar
  10. [10]
    Lin, J.J., Mamykina, L., Lindtner, S., Delajoux, G., & Strub, H.B. (2006). Fish‘n’Steps: Encouraging Physical Activity with an Interactive Computer Game, In Proceedings of the 8 th International Conference on Ubiquitous Computing: UbiComp ’06, Orange County, CA, USA, 261-78.Google Scholar
  11. [11]
    Logan, B., Healey, J., Philipose, M., Munguia-Tapia, E., Intille. S. Long-Term Evaluation of Sensing Modalities for Activity Recognition. In Proceedings of Ubicomp 2007. Innsbruck, Austria, September 2007Google Scholar
  12. [12]
    Maitland, J., Sherwood, S., Barkhuus, L., Anderson, I., Hall, M., Brown, B., Chalmers, M., & Muller, H. Increasing the Awareness of Daily Activity Levels with Pervasive Computing, it Proceedings of the 1st International Conference on Pervasive Computing Technologies for Healthcare 2006: Pervasive Health ’06, Innsbruck, Austria.Google Scholar
  13. [13]
    Mahdaviani. M. and Choudhury, T. Fast and Scalable Training of Semi-Supervised CRFs with Application to Activity Recognition. Appears in the Proceedings of NIPS 2007. December 2007.Google Scholar
  14. [14]
    Mueller, F., Agamanolis, S., Gibbs, M.R., & Vetere, F. (Apr 2008). Remote impact: shadowboxing over a distance, CHI ’08 Extended Abstracts on Human Factors in Computing Systems, Florence, Italy, 2291-6.Google Scholar
  15. [15]
    Mueller, F., Agamanolis, S., & Picard, R. (Apr 2003). Exertion Interfaces: Sports Over a Distance for Social Bonding and Fun, Proceedings of CHI ‘03, pp.561-8.Google Scholar
  16. [16]
    Mueller, F., O’Brien, S., & Thorogood, A. (Apr 2007). Jogging Over a Distance, In CHI ’07 Extended Abstracts, 1989-94.Google Scholar
  17. [17]
    Oliver, N. & Flores-Mangas, F. (Sep 2006). MPTrain: A Mobile, Music and Physiology-Based Personal Trainer, In Proceedings of MobileHCI ’06.Google Scholar
  18. [18]
    Schapire, R.E., A Brief Introduction to Boosting, Proc. 16th Int’l Joint Conf. Artificial Intelligence (ijcai 99), Morgan Kaufman, 1999, pp. 1401-1406.Google Scholar
  19. [19]
    Smith, J. R., Fishkin, K., Jiang, B., Mamishev, A., Philipose, M., Rea, A., Roy, S., Sundara-Rajan, K., RFID-Based Techniques for Human Activity Recognition. Communications of the ACM, v48, no. 9, Sep 2005.Google Scholar
  20. [20]
    Sohn, T., et al., Mobility Detection Using Everyday GSM Traces, In Proceedings of the 8 th International Conference on Ubiquitous Computing: UbiComp 2006, Orange County, CA, USA.Google Scholar
  21. [21]
    Sohn, T., Griswold, W., Scott, J., LaMarca, A., Chawathe, Y., Smith, I.E., Chen, M.,Y., Experiences with place lab: an open source toolkit for location-aware computing.ICSE 2006: 462-476.Google Scholar
  22. [22]
    Viola, P., Jones, M.: Rapid Object Detection using a Boosted Cascade of Simple Features. In: Proc. Computer Vision and Pattern Recognition (2001).Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2010

Authors and Affiliations

  1. 1.Intel ResearchSeattleUSA
  2. 2.Department of Computer ScienceDartmouth CollegeHanoverUSA

Personalised recommendations