Ontology-based sensor fusion activity recognition

  • Mohd Halim Mohd Noor
  • Zoran Salcic
  • Kevin I-Kai Wang
Original Research


This paper investigates the fusion of wearable and ambient sensors for recognizing activities of daily living in a smart home setting using ontology. The proposed approach exploits the advantages of both types of sensing to resolve uncertainties due to missing sensor data. The resulting system is able to infer activities which cannot be inferred with the single type of sensing only. The methodology of ontological modeling the wearable and ambient sensors and the fusion of contexts captured from the sensors, as well as corresponding activity is investigated and described. The proposed system is compared with a system that uses ambient sensors without wearable sensor on the internally collected and publicly available datasets. The results of the experiments show that the proposed system is more robust in handling uncertainties. It is also more capable of inferring additional information about activities, which is not possible with environment sensing only, with overall recognition accuracy of 91.5 and 90% on internal and public datasets, respectively.



The authors would like to thank Embedded System Research Group for providing sensors and devices utilized in this study. The authors would also like to thank Nazrin Muhammad (UoA) and Akshat Bisht for assisting with this work.


  1. Alirezaie M, Renoux J, Köckemann U, Kristoffersson A, Karlsson L, Blomqvist E, Tsiftes N, Voigt T, Loutfi A (2017) An ontology-based context-aware system for smart homes: e-care@home. Sensors 17:1586. Scholar
  2. Atallah L, Lo B, Ali R, King R, Yang G-Z (2009) Real-time activity classification using ambient and wearable sensors. IEEE Trans Inf Technol Biomed 13:1031–1039. Scholar
  3. Baader F, Calvanese D, McGuinness DL, Nardi D, Patel-Schneider PF (eds) (2003) The description logic handbook: theory, implementation, and applications. Cambridge University Press, New YorkMATHGoogle Scholar
  4. BakhshandehAbkenar A, Loke SW (2014) MyActivity: cloud-hosted continuous activity recognition using ontology-based stream reasoning. In: 2014 2nd IEEE international conference on mobile cloud computing, services, and engineering (MobileCloud), pp 117–126Google Scholar
  5. Chen L, Hoey J, Nugent CD, Cook DJ, Yu Z (2012) Sensor-based activity recognition. IEEE Trans Syst Man Cybern Part C Appl Rev 42:790–808. Scholar
  6. De D, Bharti P, Das SK, Chellappan S (2015) Multimodal wearable sensing for fine-grained activity recognition in healthcare. IEEE Internet Comput 19:26–35. Scholar
  7. Do TM, Loke SW, Liu F (2013) HealthyLife: an activity recognition system with smartphone using logic-based stream reasoning. In: Zheng K, Li M, Jiang H (eds) Mobile and ubiquitous systems: computing, networking, and services. 9th international conference, MobiQuitous 2012, Beijing, China, December 12–14, 2012. Revised selected papers. Springer, Berlin, Heidelberg, pp 188–199Google Scholar
  8. Förster K, Biasiucci A, Chavarriaga R, del Millán JR, Roggen D, Tröster G (2010) On the use of brain decoded signals for online user adaptive gesture recognition systems. In: Floréen P, Krüger A, Spasojevic M (eds) Pervasive computing. Springer, Berlin, pp 427–444CrossRefGoogle Scholar
  9. Gavrila DM (1999) The visual analysis of human movement: a survey. Comput Vis Image Underst 73:82–98. Scholar
  10. Ge Y, Xu B (2014) Elderly personal intention recognition by activity and context recognition in smart home. In: 2014 9th international conference on computer science education (ICCSE), pp 347–350Google Scholar
  11. Gravina R, Alinia P, Ghasemzadeh H, Fortino G (2017) Multi-sensor fusion in body sensor networks: state-of-the-art and research challenges. Inf Fusion 35:68–80. Scholar
  12. Gu T, Wang L, Wu Z, Tao X, Lu J (2011) A pattern mining approach to sensor-based human activity recognition. IEEE Trans Knowl Data Eng 23:1359–1372. Scholar
  13. Hodges MR, Pollack ME (2007) An ‘object-use fingerprint’: the use of electronic sensors for human identification. In: Krumm J, Abowd GD, Seneviratne A, Strang T (eds) UbiComp 2007: ubiquitous computing. Springer, Berlin, pp 289–303CrossRefGoogle Scholar
  14. Jia R, Liu B (2013) Human daily activity recognition by fusing accelerometer and multi-lead ECG data. In: 2013 IEEE international conference on signal processing, communication and computing (ICSPCC), pp 1–4Google Scholar
  15. Khattak AM, Truc PTH, Hung LX, Vinh LT, Dang V-H, Guan D, Pervez Z, Han M, Lee S, Lee Y-K (2011) Towards smart homes using low level sensory data. Sensors 11:11581–11604. Scholar
  16. Laukkanen P, Leskinen E, Kauppinen M, Sakari-Rantala R, Heikkinen E (2000) Health and functional capacity as predictors of community dwelling among elderly people. J Clin Epidemiol 53:257–265. Scholar
  17. Lee ML, Dey AK (2014) Sensor-based observations of daily living for aging in place. Pers Ubiquitous Comput 19:27–43. Scholar
  18. McIlwraith D, Pansiot J, Yang G-Z (2010) Wearable and ambient sensor fusion for the characterisation of human motion. In: 2010 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp 5505–5510Google Scholar
  19. Ni Q, García Hernando AB, de la Cruz IP (2015) The elderly’s independent living in smart homes: a characterization of activities and sensing infrastructure survey to facilitate services development. Sensors 15:11312–11362. Scholar
  20. Noor MHM, Salcic Z, Wang KI-K (2017) Adaptive sliding window segmentation for physical activity recognition using a single tri-axial accelerometer. Pervasive Mob Comput 38:41–59. Scholar
  21. Pansiot J, Stoyanov D, McIlwraith D, Lo BPL, Yang GZ (2007) Ambient and wearable sensor fusion for activity recognition in healthcare monitoring systems. In: Leonhardt S, Falck D-IT, Mähönen PDP (eds) 4th international workshop on wearable and implantable body sensor networks (BSN 2007). Springer, Berlin, pp 208–212Google Scholar
  22. Peeters PHF (2000) Design criteria for an automatic safety-alarm system for elderly. Technol Health Care 8:81–91Google Scholar
  23. Philipose M, Fishkin KP, Perkowitz M, Patterson DJ, Fox D, Kautz H, Hahnel D (2004) Inferring activities from interactions with objects. IEEE Pervasive Comput 3:50–57. Scholar
  24. Riboni D, Bettini C (2011) COSAR: hybrid reasoning for context-aware activity recognition. Pers Ubiquitous Comput 15:271–289. Scholar
  25. Rodríguez ND, Cuéllar MP, Lilius J, Calvo-Flores MD (2014) A survey on ontologies for human behavior recognition. ACM Comput Surv 46:43:1–43:33. Scholar
  26. Roggen D, Calatroni A, Rossi M, Holleczek T, Forster K, Troster G, Lukowicz P, Bannach D, Pirkl G, Ferscha A, Doppler J, Holzmann C, Kurz M, Holl G, Chavarriaga R, Sagha H, Bayati H, Creatura M, del JR Millan (2010) Collecting complex activity datasets in highly rich networked sensor environments. In: 2010 Seventh international conference on networked sensing systems (INSS), pp 233–240Google Scholar
  27. Roy N, Misra A, Cook D (2015) Ambient and smartphone sensor assisted ADL recognition in multi-inhabitant smart environments. J Ambient Intell Humaniz Comput 7:1–19. Scholar
  28. Sagha H, Digumarti ST, Millán J, del R, Chavarriaga, Calatroni R, Roggen A, Tröster D G (2011) Benchmarking classification techniques using the opportunity human activity dataset. In: 2011 IEEE international conference on systems, man, and cybernetics (SMC), pp 36–40Google Scholar
  29. Scalmato A, Sgorbissa A, Zaccaria R (2013) Describing and recognizing patterns of events in smart environments with description logic. IEEE Trans Cybern 43:1882–1897. Scholar
  30. Shoaib M, Bosch S, Incel OD, Scholten H, Havinga PJM (2014) Fusion of smartphone motion sensors for physical activity recognition. Sensors 14:10146–10176. Scholar
  31. Steigmiller A, Glimm B, Liebig T (2014) Coupling tableau algorithms for expressive description logics with completion-based saturation procedures. In: Demri S, Kapur D, Weidenbach C (eds) Automated Reasoning. Proceedings of 7th international joint conference, IJCAR 2014, held as part of the Vienna summer of logic, VSL 2014, Vienna, Austria, July 19–22, 2014. Springer, Cham, pp 449–463Google Scholar
  32. Sun H, Fan W, Shen W, Xiao T (2013) Ontology fusion in high-level-architecture-based collaborative engineering environments. IEEE Trans Syst Man Cybern Syst 43:2–13. Scholar
  33. Sung M, DeVaul R, Jimenez S, Gips J, Pentland A (2004) Shiver motion and core body temperature classification for wearable soldier health monitoring systems. In: Eighth international symposium on wearable computers, 2004. ISWC 2004, pp 192–193Google Scholar
  34. Turaga P, Chellappa R, Subrahmanian VS, Udrea O (2008) Machine recognition of human activities: a survey. IEEE Trans Circuits Syst Video Technol 18:1473–1488. Scholar
  35. Valle ED, Ceri S, Harmelen FV, Fensel D (2009) It’s a streaming world! Reasoning upon rapidly changing information. IEEE Intell Syst 24:83–89. Scholar
  36. Villalonga C, Pomares H, Rojas I, Banos O (2017) MIMU-wear: ontology-based sensor selection for real-world wearable activity recognition. Neurocomputing 250:76–100. Scholar
  37. Wang C, Cao L, Chi CH (2015) Formalization and verification of group behavior interactions. IEEE Trans Syst Man Cybern Syst 45:1109–1124. Scholar
  38. Wannenburg J, Malekian R (2016) Physical activity recognition from smartphone accelerometer data for user context awareness sensing. IEEE Trans Syst Man Cybern Syst PP:1–8.
  39. WHO (2014) WHO, facts about ageing. In: WHO. Accessed 18 Sept 2017
  40. Wongpatikaseree K, Ikeda M, Buranarach M, Supnithi T, Lim AO, Tan Y (2012) Activity recognition using context-aware infrastructure ontology in smart home domain. In: 2012 Seventh international conference on knowledge, information and creativity support systems (KICSS), pp 50–57Google Scholar
  41. Wu K, Haarslev V (2014) Parallel OWL reasoning: merge classification. In: Kim W, Ding Y, Kim H-G (eds) Semantic technology. 3rd joint international conference, JIST 2013, Seoul, South Korea, November 28–30, 2013, Revised Selected Papers. Springer, Cham, pp 211–227Google Scholar
  42. Wu J, Osuntogun A, Choudhury T, Philipose M, Rehg JM (2007) A scalable approach to activity recognition based on object use. In: 2007 IEEE 11th international conference on computer vision, pp 1–8Google Scholar
  43. Yilmaz A, Javed O, Shah M (2006) Object tracking: a survey. ACM Comput Surv. Scholar
  44. Yoshihiro T, Masako K-P, Noriaki K, Jukai M, Miwa H, Yasuko K, Ota Jun (2013) Recognition of nursing activity with accelerometers and RFID. Kybernetes 42:1059–1071. Scholar
  45. Zgheib R, Nicola AD, Villani ML, Conchon E, Bastide R (2017) A flexible architecture for cognitive sensing of activities in ambient assisted living. In: 2017 IEEE 26th international conference on enabling technologies: infrastructure for collaborative enterprises (WETICE), pp 284–289Google Scholar
  46. Zhou F, Jiao JR, Chen S, Zhang D (2011) A case-driven ambient intelligence system for elderly in-home assistance applications. IEEE Trans Syst Man Cybern Part C Appl Rev 41:179–189. Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Faculty of Electrical EngineeringUniversiti Teknologi MARAPermatang PauhMalaysia
  2. 2.Department of Electrical and Computer EngineeringThe University of AucklandAucklandNew Zealand

Personalised recommendations