Hybrid Convolutional Neural Network Ensemble for Activity Recognition in Mobile Phones

  • Jyh-Huah Chan
  • Hui-Juin Lim
  • Ngoc-Son Hoang
  • Jeong-Hoon Lim
  • Khang Nguyen
  • Binh P. NguyenEmail author
  • Chee-Kong Chui
  • Matthew Chin-Heng Chua
Part of the Studies in Computational Intelligence book series (SCI, volume 899)


The increasing importance of human activity recognition in Ambient Assisted Living systems (AAL) has brought a tremendous growth in research efforts in the field. Amongst these, the identification of activities of daily living using sensors available in mobile devices has emerged as one of the most interesting goals for AAL systems. With the development of deep learning algorithms in recent years, there has been increasing interest in hybrid models, which are able to perform as well as traditional machine learning models and yet entail the self-learning capabilities of deep learning at the same time. Using the time series data obtained from the MotionSense dataset, this paper introduces the concept of converting time series signals to images. The paper then proposes a novel approach for activity recognition by fusing a variety of traditional machine learning models with a deep convolutional neural network via a majority voting ensemble. The proposed method produced better results than the traditional machine learning models.


Activity recognition Recurrence plots Ensemble Convolutional neural networks Machine learning Deep learning 



This research is supported by the Singapore Ministry of Health’s National Medical Research Council under its Enabling Innovation Grant, Grant No: NMRC/ EIG06/2017.


  1. 1.
    Belbachir, A.N., Drobics, M., Marschitz, W.: Ambient assisted living for ageing well - an overview. e & i Elektrotechnik und Informationstechnik 127(7), 200–205 (2010)CrossRefGoogle Scholar
  2. 2.
    Brdiczka, O., Crowley, J.L., Reignier, P.: Learning situation models in a smart home. IEEE Trans. Syst. Man Cybern. Part B (Cybern.) 39(1), 56–63 (2009)CrossRefGoogle Scholar
  3. 3.
    Brdiczka, O., Reignier, P., Crowley, J.L.: Detecting individual activities from video in a smart home. In: Apolloni, B., Howlett, R.J., Jain, L. (eds.) Knowl. Based Intell. Inf. Eng. Syst., pp. 363–370. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  4. 4.
    Chen, M., Huang, B., Xu, Y.: Human abnormal gait modeling via hidden Markov model. In: Proceedings of the 2007 International Conference on Information Acquisition, pp. 517–522 (2007)Google Scholar
  5. 5.
    Chen, X., Nguyen, B.P., Chui, C.K., Ong, S.H.: Automated brain tumor segmentation using kernel dictionary learning and superpixel-level features. In: Proceedings of the International Conference on Systems, Man, and Cybernetics, SMC 2016, pp. 2547–2552, Budapesh, Hungary, IEEE. 9–12 October 2016Google Scholar
  6. 6.
    Chen, X., Nguyen, B.P., Chui, C.-K., Ong, S.-H.: An automatic framework for multi-label brain tumor segmentation based on kernel sparse representation. Acta Polytechnica Hungarica 14(1), 25–43 (2017)Google Scholar
  7. 7.
    Chen, X., Nguyen, B.P., Chui, C.-K., Ong, S.-H.: Reworking multilabel brain tumor segmentation - an automated framework using structured kernel sparse representation. IEEE Syst. Man Cybern. Mag. 3(2), 18–22 (2017)CrossRefGoogle Scholar
  8. 8.
    Cook, D.J., Schmitter-Edgecombe, M.: Assessing the quality of activities in a smart environment. Methods Inf. Med. 48, 480–485 (2009)CrossRefGoogle Scholar
  9. 9.
    Donahue, J.: Long-term recurrent convolutional networks for visual recognition and description. IEEE Trans. Pattern Anal. Mach. Intell. 39(4), 677–691 (2017)CrossRefGoogle Scholar
  10. 10.
    Eckmann, J.P., Kamphorst, S.O., Ruelle, D.: Recurrence plots of dynamical systems. Europhys. Lett. 4(9), 973–977 (1987)CrossRefGoogle Scholar
  11. 11.
    Fernandez-Lopez, P., Sanchez-Casanova, J., Tirado-Martín, P., Liu-Jimenez, J.: Optimizing resources on smartphone gait recognition. In: Proceedings of the IEEE International Joint Conference on Biometrics, IJCB 2017, pp. 31–36, October 2017Google Scholar
  12. 12.
    Graves, A., Mohamed, A.R., Hinton, G.E.: Speech recognition with deep recurrent neural networks. In: Proceedings of the International Conference on Acoustics, Speech and Signal Processing, ICASSP 2013, pp. 6645–6649 (2013)Google Scholar
  13. 13.
    Hatami, N., Gavet, Y., Debayle, J.: Classification of time-series images using deep convolutional neural networks. In: Verikas, A., Radeva, P., Nikolaev, D., Zhou, J., (eds.) Proceedings of the 10th International Conference on Machine Vision, ICMV 2017, vol. 10696, pp. 242–249. International Society for Optics and Photonics, SPIE (2018)Google Scholar
  14. 14.
    Hinton, G.E., Osindero, S., Teh, Y.-W.: A fast learning algorithm for deep belief nets. Neural Comput. 18(7), 1527–1554 (2006)MathSciNetzbMATHCrossRefGoogle Scholar
  15. 15.
    Jain, A.K., Duin, R.P.W., Mao, J.: Statistical pattern recognition: a review. IEEE Trans. Pattern Anal. Mach. Intell. 22(1), 4–37 (2000)CrossRefGoogle Scholar
  16. 16.
    Johnston, A.H., Weiss, G.M.: Smartwatch-based biometric gait recognition. In: Proceedings of the 7th IEEE International Conference on Biometrics Theory, Applications and Systems, BTAS 2015, pp. 1–6, September 2015Google Scholar
  17. 17.
    Juefei-Xu, F., Bhagavatula, C., Jaech, A., Prasad, U., Savvides, M.: Gait-ID on the move: pace independent human identification using cell phone accelerometer dynamics. In: Proceedings of the 5th IEEE International Conference on Biometrics: Theory, Applications and Systems BTAS 2012, pp. 8–15, September 2012Google Scholar
  18. 18.
    van Kasteren, T., Krose, B.: Bayesian activity recognition in residence for elders. In: Proceedings of the 3rd IET International Conference on Intelligent Environments, pp. 209–212, September 2007Google Scholar
  19. 19.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Mag. Commun. ACM 60(6), 84–90 (2017)CrossRefGoogle Scholar
  20. 20.
    Kwapisz, J.R., Weiss, G.M., Moore, S.A.: Cell phone-based biometric identification. In: Proceedings of the 4th IEEE International Conference on Biometrics: Theory, Applications and Systems BTAS 2010, pp. 1–7, September 2010Google Scholar
  21. 21.
    Malekzadeh, M., Clegg, R.G., Cavallaro, A., Haddadi, H.: Protecting sensory data against sensitive inferences. In: Proceedings of the 1st Workshop on Privacy by Design in Distributed Systems, pp. 2:1–2:6, New York, NY, USA. ACM (2018)Google Scholar
  22. 22.
    Mannini, A., Sabatini, A.M.: Gait phase detection and discrimination between walking-jogging activities using hidden Markov models applied to foot motion data from a gyroscope. Gait Posture 36(4), 657–661 (2012)CrossRefGoogle Scholar
  23. 23.
    Mannini, A., Sabatini, A.M.: A smartphone-centered wearable sensor network for fall risk assessment in the elderly. In: Proceedings of the 10th EAI International Conference on Body Area Networks, BodyNets 2015, pp. 167–172 (2015)Google Scholar
  24. 24.
    Mannini, A., Sabatini, A.M., Intille, S.S.: Accelerometry-based recognition of the placement sites of a wearable sensor. Pervasive Mobile Comput. 21, 62–74 (2015)CrossRefGoogle Scholar
  25. 25.
    Maurer, U., Rowe, A., Smailagic, A., Siewiorek, D.: Location and activity recognition using eWatch: a wearable sensor platform. In: Cai, Y., Abascal, J. (eds.) Ambient Intelligence in Everyday Life: Foreword by Emile Aarts, pp. 86–102. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  26. 26.
    Nguyen, B.P., Heemskerk, H., So, P.T.C., Tucker-Kellogg, L.: Superpixel-based segmentation of muscle fibers in multi-channel microscopy. BMC Syst. Biol. 10(5), 124 (2016)CrossRefGoogle Scholar
  27. 27.
    Nguyen, B.P., Tay, W.-L., Chui, C.-K.: Robust biometric recognition from palm depth images for gloved hands. IEEE Trans. Human-Mach. Syst. 45(6), 799–804 (2015)CrossRefGoogle Scholar
  28. 28.
    Ravi, N., Dandekar, N., Mysore, P., Littman, M.L.: Activity recognition from accelerometer data. In: Proceedings of the 17th Conference on Innovative Applications of Artificial Intelligence - Volume 3, IAAI 2005, pp. 1541–1546. AAAI Press (2005)Google Scholar
  29. 29.
    Rida, I., Almaadeed, N., Almaadeed, S.: Robust gait recognition: a comprehensive survey. IET Biometrics 8(1), 14–28 (2019)CrossRefGoogle Scholar
  30. 30.
    Stolze, H., Klebe, S., Zechlin, C., Baecker, C., Friege, L., Deuschl, G.: Falls in frequent neurological diseases: Prevalence, risk factors and aetiology. J. Neurol. 251(1), 79–84 (2004)CrossRefGoogle Scholar
  31. 31.
    Sun, B., Wang, Y., Banda, J.: Gait characteristic analysis and identification based on the iPhone’s accelerometer and gyrometer. Sensors 14(9), 17037–17054 (2014)CrossRefGoogle Scholar
  32. 32.
    Taborri, J., Rossi, S., Palermo, E., Patane, F., Cappa, P.: A novel HMM distributed classifier for the detection of gait phases by means of a wearable inertial sensor network. Sensors 14(9), 16212–16234 (2014)CrossRefGoogle Scholar
  33. 33.
    Trung, N.T., Makihara, Y., Nagahara, H., Mukaigawa, Y., Yagi, Y.: Performance evaluation of gait recognition using the largest inertial sensor-based gait database. In: Proceedings of the 5th IAPR International Conference on Biometrics, ICB 2012, pp. 360–366, March 2012Google Scholar
  34. 34.
    Wang, P., Li, W., Li, C., Hou, Y.: Action recognition based on joint trajectory maps with convolutional neural networks. Knowl. Based Syst. 158, 43–53 (2018)CrossRefGoogle Scholar
  35. 35.
    Bo Yang, J., Nguyen, M.N., San, P.P., Li, X.L., Krishnaswamy, S.: Deep convolutional neural networks on multichannel time series for human activity recognition. In: Proceedings of the 24th International Conference on Artificial Intelligence, IJCAI 2015, pp. 3995–4001. AAAI Press (2015)Google Scholar
  36. 36.
    Yang, Q.: Activity recognition: linking low-level sensors to high-level intelligence. In: Proceedings of the 21st International Joint Conference on Artificial Intelligence, IJCAI 2009, pp. 20–25, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc (2009)Google Scholar
  37. 37.
    Yang, X., Zhang, C., Tian, Y.: Recognizing actions using depth motion maps-based histograms of oriented gradients. In: Proceedings of the 20th ACM International Conference on Multimedia, MM 2012, pp. 1057–1060, New York, NY, USA. ACM (2012)Google Scholar
  38. 38.
    Yilmaz, A., Javed, O., Shah, M.: Object tracking: a survey. ACM Computing Surveys 38(4) (2006)Google Scholar
  39. 39.
    Yoneda, K., Weiss, G.M.: Mobile sensor-based biometrics using common daily activities. In: Proceedings of the 8th IEEE Annual Ubiquitous Computing, Electronics and Mobile Communication Conference, UEMCON 2017, pp. 584–590, October 2017Google Scholar

Copyright information

© The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2021

Authors and Affiliations

  • Jyh-Huah Chan
    • 1
  • Hui-Juin Lim
    • 1
  • Ngoc-Son Hoang
    • 1
  • Jeong-Hoon Lim
    • 2
  • Khang Nguyen
    • 3
  • Binh P. Nguyen
    • 4
    Email author
  • Chee-Kong Chui
    • 5
  • Matthew Chin-Heng Chua
    • 1
  1. 1.Institute of Systems ScienceNational University of SingaporeSingaporeSingapore
  2. 2.Division of NeurologyUniversity Medicine Cluster, National University HospitalSingaporeSingapore
  3. 3.IBM VietnamHanoiVietnam
  4. 4.School of Mathematics and StatisticsVictoria University of WellingtonWellingtonNew Zealand
  5. 5.Department of Mechanical EngineeringNational University of SingaporeSingaporeSingapore

Personalised recommendations