Skip to main content
SpringerLink
Log in

Ontology-based sensor fusion activity recognition

  • Original Research
  • Published:
Journal of Ambient Intelligence and Humanized Computing Aims and scope Submit manuscript

Abstract

This paper investigates the fusion of wearable and ambient sensors for recognizing activities of daily living in a smart home setting using ontology. The proposed approach exploits the advantages of both types of sensing to resolve uncertainties due to missing sensor data. The resulting system is able to infer activities which cannot be inferred with the single type of sensing only. The methodology of ontological modeling the wearable and ambient sensors and the fusion of contexts captured from the sensors, as well as corresponding activity is investigated and described. The proposed system is compared with a system that uses ambient sensors without wearable sensor on the internally collected and publicly available datasets. The results of the experiments show that the proposed system is more robust in handling uncertainties. It is also more capable of inferring additional information about activities, which is not possible with environment sensing only, with overall recognition accuracy of 91.5 and 90% on internal and public datasets, respectively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Notes

  1. Hermit OWL reasoner http://www.hermit-reasoner.com/.

  2. The activity ontology is available at the address http://halim.readismed.com/download/activity-ontology.

References

  • Alirezaie M, Renoux J, Köckemann U, Kristoffersson A, Karlsson L, Blomqvist E, Tsiftes N, Voigt T, Loutfi A (2017) An ontology-based context-aware system for smart homes: e-care@home. Sensors 17:1586. https://doi.org/10.3390/s17071586

    Article  Google Scholar 

  • Atallah L, Lo B, Ali R, King R, Yang G-Z (2009) Real-time activity classification using ambient and wearable sensors. IEEE Trans Inf Technol Biomed 13:1031–1039. https://doi.org/10.1109/TITB.2009.2028575

    Article  Google Scholar 

  • Baader F, Calvanese D, McGuinness DL, Nardi D, Patel-Schneider PF (eds) (2003) The description logic handbook: theory, implementation, and applications. Cambridge University Press, New York

    MATH  Google Scholar 

  • BakhshandehAbkenar A, Loke SW (2014) MyActivity: cloud-hosted continuous activity recognition using ontology-based stream reasoning. In: 2014 2nd IEEE international conference on mobile cloud computing, services, and engineering (MobileCloud), pp 117–126

  • Chen L, Hoey J, Nugent CD, Cook DJ, Yu Z (2012) Sensor-based activity recognition. IEEE Trans Syst Man Cybern Part C Appl Rev 42:790–808. https://doi.org/10.1109/TSMCC.2012.2198883

    Article  Google Scholar 

  • De D, Bharti P, Das SK, Chellappan S (2015) Multimodal wearable sensing for fine-grained activity recognition in healthcare. IEEE Internet Comput 19:26–35. https://doi.org/10.1109/MIC.2015.72

    Article  Google Scholar 

  • Do TM, Loke SW, Liu F (2013) HealthyLife: an activity recognition system with smartphone using logic-based stream reasoning. In: Zheng K, Li M, Jiang H (eds) Mobile and ubiquitous systems: computing, networking, and services. 9th international conference, MobiQuitous 2012, Beijing, China, December 12–14, 2012. Revised selected papers. Springer, Berlin, Heidelberg, pp 188–199

    Google Scholar 

  • Förster K, Biasiucci A, Chavarriaga R, del Millán JR, Roggen D, Tröster G (2010) On the use of brain decoded signals for online user adaptive gesture recognition systems. In: Floréen P, Krüger A, Spasojevic M (eds) Pervasive computing. Springer, Berlin, pp 427–444

    Chapter  Google Scholar 

  • Gavrila DM (1999) The visual analysis of human movement: a survey. Comput Vis Image Underst 73:82–98. https://doi.org/10.1006/cviu.1998.0716

    Article  MATH  Google Scholar 

  • Ge Y, Xu B (2014) Elderly personal intention recognition by activity and context recognition in smart home. In: 2014 9th international conference on computer science education (ICCSE), pp 347–350

  • Gravina R, Alinia P, Ghasemzadeh H, Fortino G (2017) Multi-sensor fusion in body sensor networks: state-of-the-art and research challenges. Inf Fusion 35:68–80. https://doi.org/10.1016/j.inffus.2016.09.005

    Article  Google Scholar 

  • Gu T, Wang L, Wu Z, Tao X, Lu J (2011) A pattern mining approach to sensor-based human activity recognition. IEEE Trans Knowl Data Eng 23:1359–1372. https://doi.org/10.1109/TKDE.2010.184

    Article  Google Scholar 

  • Hodges MR, Pollack ME (2007) An ‘object-use fingerprint’: the use of electronic sensors for human identification. In: Krumm J, Abowd GD, Seneviratne A, Strang T (eds) UbiComp 2007: ubiquitous computing. Springer, Berlin, pp 289–303

    Chapter  Google Scholar 

  • Jia R, Liu B (2013) Human daily activity recognition by fusing accelerometer and multi-lead ECG data. In: 2013 IEEE international conference on signal processing, communication and computing (ICSPCC), pp 1–4

  • Khattak AM, Truc PTH, Hung LX, Vinh LT, Dang V-H, Guan D, Pervez Z, Han M, Lee S, Lee Y-K (2011) Towards smart homes using low level sensory data. Sensors 11:11581–11604. https://doi.org/10.3390/s111211581

    Article  Google Scholar 

  • Laukkanen P, Leskinen E, Kauppinen M, Sakari-Rantala R, Heikkinen E (2000) Health and functional capacity as predictors of community dwelling among elderly people. J Clin Epidemiol 53:257–265. https://doi.org/10.1016/S0895-4356(99)00178-X

    Article  Google Scholar 

  • Lee ML, Dey AK (2014) Sensor-based observations of daily living for aging in place. Pers Ubiquitous Comput 19:27–43. https://doi.org/10.1007/s00779-014-0810-3

    Article  Google Scholar 

  • McIlwraith D, Pansiot J, Yang G-Z (2010) Wearable and ambient sensor fusion for the characterisation of human motion. In: 2010 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp 5505–5510

  • Ni Q, García Hernando AB, de la Cruz IP (2015) The elderly’s independent living in smart homes: a characterization of activities and sensing infrastructure survey to facilitate services development. Sensors 15:11312–11362. https://doi.org/10.3390/s150511312

    Article  Google Scholar 

  • Noor MHM, Salcic Z, Wang KI-K (2017) Adaptive sliding window segmentation for physical activity recognition using a single tri-axial accelerometer. Pervasive Mob Comput 38:41–59. https://doi.org/10.1016/j.pmcj.2016.09.009

    Article  Google Scholar 

  • Pansiot J, Stoyanov D, McIlwraith D, Lo BPL, Yang GZ (2007) Ambient and wearable sensor fusion for activity recognition in healthcare monitoring systems. In: Leonhardt S, Falck D-IT, Mähönen PDP (eds) 4th international workshop on wearable and implantable body sensor networks (BSN 2007). Springer, Berlin, pp 208–212

  • Peeters PHF (2000) Design criteria for an automatic safety-alarm system for elderly. Technol Health Care 8:81–91

    Article  Google Scholar 

  • Philipose M, Fishkin KP, Perkowitz M, Patterson DJ, Fox D, Kautz H, Hahnel D (2004) Inferring activities from interactions with objects. IEEE Pervasive Comput 3:50–57. https://doi.org/10.1109/MPRV.2004.7

    Article  Google Scholar 

  • Riboni D, Bettini C (2011) COSAR: hybrid reasoning for context-aware activity recognition. Pers Ubiquitous Comput 15:271–289. https://doi.org/10.1007/s00779-010-0331-7

    Article  Google Scholar 

  • Rodríguez ND, Cuéllar MP, Lilius J, Calvo-Flores MD (2014) A survey on ontologies for human behavior recognition. ACM Comput Surv 46:43:1–43:33. https://doi.org/10.1145/2523819

    Article  Google Scholar 

  • Roggen D, Calatroni A, Rossi M, Holleczek T, Forster K, Troster G, Lukowicz P, Bannach D, Pirkl G, Ferscha A, Doppler J, Holzmann C, Kurz M, Holl G, Chavarriaga R, Sagha H, Bayati H, Creatura M, del JR Millan (2010) Collecting complex activity datasets in highly rich networked sensor environments. In: 2010 Seventh international conference on networked sensing systems (INSS), pp 233–240

  • Roy N, Misra A, Cook D (2015) Ambient and smartphone sensor assisted ADL recognition in multi-inhabitant smart environments. J Ambient Intell Humaniz Comput 7:1–19. https://doi.org/10.1007/s12652-015-0294-7

    Article  Google Scholar 

  • Sagha H, Digumarti ST, Millán J, del R, Chavarriaga, Calatroni R, Roggen A, Tröster D G (2011) Benchmarking classification techniques using the opportunity human activity dataset. In: 2011 IEEE international conference on systems, man, and cybernetics (SMC), pp 36–40

  • Scalmato A, Sgorbissa A, Zaccaria R (2013) Describing and recognizing patterns of events in smart environments with description logic. IEEE Trans Cybern 43:1882–1897. https://doi.org/10.1109/TSMCB.2012.2234739

    Article  Google Scholar 

  • Shoaib M, Bosch S, Incel OD, Scholten H, Havinga PJM (2014) Fusion of smartphone motion sensors for physical activity recognition. Sensors 14:10146–10176. https://doi.org/10.3390/s140610146

    Article  Google Scholar 

  • Steigmiller A, Glimm B, Liebig T (2014) Coupling tableau algorithms for expressive description logics with completion-based saturation procedures. In: Demri S, Kapur D, Weidenbach C (eds) Automated Reasoning. Proceedings of 7th international joint conference, IJCAR 2014, held as part of the Vienna summer of logic, VSL 2014, Vienna, Austria, July 19–22, 2014. Springer, Cham, pp 449–463

    Chapter  Google Scholar 

  • Sun H, Fan W, Shen W, Xiao T (2013) Ontology fusion in high-level-architecture-based collaborative engineering environments. IEEE Trans Syst Man Cybern Syst 43:2–13. https://doi.org/10.1109/TSMCA.2012.2190138

    Article  Google Scholar 

  • Sung M, DeVaul R, Jimenez S, Gips J, Pentland A (2004) Shiver motion and core body temperature classification for wearable soldier health monitoring systems. In: Eighth international symposium on wearable computers, 2004. ISWC 2004, pp 192–193

  • Turaga P, Chellappa R, Subrahmanian VS, Udrea O (2008) Machine recognition of human activities: a survey. IEEE Trans Circuits Syst Video Technol 18:1473–1488. https://doi.org/10.1109/TCSVT.2008.2005594

    Article  Google Scholar 

  • Valle ED, Ceri S, Harmelen FV, Fensel D (2009) It’s a streaming world! Reasoning upon rapidly changing information. IEEE Intell Syst 24:83–89. https://doi.org/10.1109/MIS.2009.125

    Article  Google Scholar 

  • Villalonga C, Pomares H, Rojas I, Banos O (2017) MIMU-wear: ontology-based sensor selection for real-world wearable activity recognition. Neurocomputing 250:76–100. https://doi.org/10.1016/j.neucom.2016.09.125

    Article  Google Scholar 

  • Wang C, Cao L, Chi CH (2015) Formalization and verification of group behavior interactions. IEEE Trans Syst Man Cybern Syst 45:1109–1124. https://doi.org/10.1109/TSMC.2015.2399862

    Article  Google Scholar 

  • Wannenburg J, Malekian R (2016) Physical activity recognition from smartphone accelerometer data for user context awareness sensing. IEEE Trans Syst Man Cybern Syst PP:1–8. https://doi.org/10.1109/TSMC.2016.2562509

  • WHO (2014) WHO, facts about ageing. In: WHO. http://www.who.int/ageing/about/facts/en/. Accessed 18 Sept 2017

  • Wongpatikaseree K, Ikeda M, Buranarach M, Supnithi T, Lim AO, Tan Y (2012) Activity recognition using context-aware infrastructure ontology in smart home domain. In: 2012 Seventh international conference on knowledge, information and creativity support systems (KICSS), pp 50–57

  • Wu K, Haarslev V (2014) Parallel OWL reasoning: merge classification. In: Kim W, Ding Y, Kim H-G (eds) Semantic technology. 3rd joint international conference, JIST 2013, Seoul, South Korea, November 28–30, 2013, Revised Selected Papers. Springer, Cham, pp 211–227

    Google Scholar 

  • Wu J, Osuntogun A, Choudhury T, Philipose M, Rehg JM (2007) A scalable approach to activity recognition based on object use. In: 2007 IEEE 11th international conference on computer vision, pp 1–8

  • Yilmaz A, Javed O, Shah M (2006) Object tracking: a survey. ACM Comput Surv. https://doi.org/10.1145/1177352.1177355

    Article  Google Scholar 

  • Yoshihiro T, Masako K-P, Noriaki K, Jukai M, Miwa H, Yasuko K, Ota Jun (2013) Recognition of nursing activity with accelerometers and RFID. Kybernetes 42:1059–1071. https://doi.org/10.1108/K-02-2013-0023

    Article  Google Scholar 

  • Zgheib R, Nicola AD, Villani ML, Conchon E, Bastide R (2017) A flexible architecture for cognitive sensing of activities in ambient assisted living. In: 2017 IEEE 26th international conference on enabling technologies: infrastructure for collaborative enterprises (WETICE), pp 284–289

  • Zhou F, Jiao JR, Chen S, Zhang D (2011) A case-driven ambient intelligence system for elderly in-home assistance applications. IEEE Trans Syst Man Cybern Part C Appl Rev 41:179–189. https://doi.org/10.1109/TSMCC.2010.2052456

    Article  Google Scholar 

Download references

Acknowledgements

The authors would like to thank Embedded System Research Group for providing sensors and devices utilized in this study. The authors would also like to thank Nazrin Muhammad (UoA) and Akshat Bisht for assisting with this work.

Author information

Authors and Affiliations

  1. Faculty of Electrical Engineering, Universiti Teknologi MARA, Permatang Pauh, Penang, Malaysia

    Mohd Halim Mohd Noor

  2. Department of Electrical and Computer Engineering, The University of Auckland, Auckland, New Zealand

    Zoran Salcic & Kevin I-Kai Wang

Authors
  1. Mohd Halim Mohd Noor
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zoran Salcic
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kevin I-Kai Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mohd Halim Mohd Noor.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Noor, M.H.M., Salcic, Z. & Wang, K.IK. Ontology-based sensor fusion activity recognition. J Ambient Intell Human Comput 11, 3073–3087 (2020). https://doi.org/10.1007/s12652-017-0668-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12652-017-0668-0

Search

Navigation