Skip to main content
SpringerLink
Log in
Book cover

Proceedings of SAI Intelligent Systems Conference

IntelliSys 2018: Intelligent Systems and Applications pp 101–119Cite as

PerceptionNet: A Deep Convolutional Neural Network for Late Sensor Fusion

  • Conference paper
  • First Online:

Part of the book series: Advances in Intelligent Systems and Computing ((AISC,volume 868))

Abstract

Human Activity Recognition (HAR) based on motion sensors has drawn a lot of attention over the last few years, since perceiving the human status enables context-aware applications to adapt their services on users’ needs. However, motion sensor fusion and feature extraction have not reached their full potentials, remaining still an open issue. In this paper, we introduce PerceptionNet, a deep Convolutional Neural Network (CNN) that applies a late 2D convolution to multimodal time-series sensor data, in order to extract automatically efficient features for HAR. We evaluate our approach on two public available HAR datasets to demonstrate that the proposed model fuses effectively multimodal sensors and improves the performance of HAR. In particular, PerceptionNet surpasses the performance of state-of-the-art HAR methods based on: (1) features extracted from humans, (2) deep CNNs exploiting early fusion approaches, and (3) Long Short-Term Memory (LSTM), by an average accuracy of more than 3%.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   169.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  1. Bosems, S., van Sinderen, M.: Model-driven development for user-centric well-being support from dynamic well-being domain models to context-aware applications. In: 3rd International Conference on Model-Driven Engineering and Software Development (MODELSWARD), pp. 425–432 (2015)

    Google Scholar 

  2. Ongenae, F., Claeys, M., Dupont, T., Kerckhove, W., Verhoeve, P., Dhaene, T., Turck, F.D.: A probabilistic ontology-based platform for self-learning context-aware healthcare applications. Expert Syst. Appl. 40(18), 7629–7646 (2013)

    Article  Google Scholar 

  3. Seo, D.W., Kim, H., Kim, J.S., Lee, J.Y.: Hybrid reality-based user experience and evaluation of a context-aware smart home. Comput. Ind. 76, 11–23 (2016)

    Article  Google Scholar 

  4. Li, W., Joshi, A., Finin, T.: SVM-CASE: An SVM-based context aware security framework for vehicular ad-hoc networks. In: IEEE 82nd Vehicular Technology Conference (VTC2015-Fall), pp. 1–5 (2015)

    Google Scholar 

  5. Patrikakis, Ch.Z., Kogias, D.G., Loukas, G., Filippoupolitis, A., Oliff, W., Rahman, S.S., Sorace, S., La Mattina, E., Quercia, E.: On the successful deployment of community policing services the TRILLION project case. In: IEEE International Conference on Consumer Electronics (ICCE 2018) (2018)

    Google Scholar 

  6. Abolfazli, S., Sanaei, Z., Gani, A., Xia, F., Yang, L.T.: Rich mobile applications: genesis, taxonomy, and open issues. J. Netw. Comput. Appl. 40, 345–362 (2014)

    Article  Google Scholar 

  7. Schilit, B.N., Theimer, M.M.: Disseminating active map information to mobile hosts. IEEE Netw. 8(5), 22–32 (1994)

    Article  Google Scholar 

  8. Dey, A.K., Abowd, G.D., Salber, D.: A conceptual framework and a toolkit for supporting the rapid prototyping of context-aware applications. Hum.-Comput. Interact. 16, 97–166 (2001)

    Article  Google Scholar 

  9. Figo, D., Diniz, P.C., Ferreira, D.R., Cardoso, J.M.P.: Preprocessing techniques for context recognition from accelerometer data. Pers. Ubiquitous Comput. 14, 645–662 (2010)

    Article  Google Scholar 

  10. LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521, 436–444 (2015)

    Article  Google Scholar 

  11. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)

    Google Scholar 

  12. Graves, A., Mohamed, A., Hinton, G.: Speech recognition with deep recurrent neural networks. In: Proceedings of ICASSP 2013, Vancouver, Canada, May 2013

    Google Scholar 

  13. Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C.D., Ng, A.Y., Potts, C.: Recursive deep models for semantic compositionality over a sentiment treebank. In: Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631–1642 (2013)

    Google Scholar 

  14. Ronao, C.A., Cho, S.-B.: Human activity recognition with smartphone sensors using deep learning neural networks. Expert Syst. Appl. 59, 235–244 (2016)

    Article  Google Scholar 

  15. Ordóñez, F.J., Roggen, D.: Deep convolutional and LSTM recurrent neural networks for multimodal wearable activity recognition. Sensors 16(1), 1–25 (2016)

    Article  Google Scholar 

  16. Reiss, A., Stricker, D.: Creating and benchmarking a new dataset for physical activity monitoring. In: The 5th Workshop on Affect and Behaviour Related Assistance (ABRA) (2012)

    Google Scholar 

  17. Kasnesis, P., Patrikakis, ChZ, Venieris, I.S.: Changing the game of mobile data analysis with deep learning. IEEE ITPro Mag. 19(3), 17–23 (2017)

    Google Scholar 

  18. Vincent, P., Larochelle, H., Bengio, Y., Manzagol, P.A.: Extracting and composing robust features with denoising autoencoders. In: Proceedings of the Twenty-Fifth International Conference on Machine Learning (ICML 2008), pp. 1096–1103. ACM (2008)

    Google Scholar 

  19. Plötz, T., Hammerla, N.Y., Olivier, P.: Feature learning for activity recognition in ubiquitous computing. In: Proceedings of the Twenty-Second IJCAI, vol. 2, pp. 1729–1734. AAAI Press (2011)

    Google Scholar 

  20. Hinton, G., Osindero, S., Teh, Y.-W.: A fast learning algorithm for deep belief nets. Neural Comput. 18(7), 1527–1554 (2006)

    Article  MathSciNet  Google Scholar 

  21. Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks. Science 313, 504–507 (2006)

    Article  MathSciNet  Google Scholar 

  22. Vollmer, C., Gross, H.-M., Eggert, J.P.: Learning features for activity recognition with shift-invariant sparse coding. In: Artificial Neural Networks and Machine Learning–ICANN, pp. 367–374 (2013)

    Chapter  Google Scholar 

  23. Lee, H., Ekanadham, C., Ng, A.Y.: Sparse deep belief net model for visual area V2. In: Advances in Neural Information Processing Systems (NIPS), vol. 20 (2008)

    Google Scholar 

  24. Zhang, Y.-D., Zhang, Y., Hou, X.-X., Chen, H., Wang, S.H.: Seven-layer deep neural network based on sparse autoencoder for voxelwise detection of cerebral microbleed. In: Multimedia Tools and Applications, pp. 1–18 (2017)

    Google Scholar 

  25. Jia, W., Yang, M.: Wang: Three-category classification of magnetic resonance hearing loss images based on deep autoencoder. J. Med. Syst. 41(10), 165 (2017)

    Article  Google Scholar 

  26. Abdel-Hamid, O., Mohamed, A.-R., Jiang, H., Penn, G.: Applying convolutional neural networks concepts to hybrid nn-hmm model for speech recognition. In: Acoustics, Speech and Signal Processing (ICASSP), pp. 4277–4280 (2012)

    Google Scholar 

  27. Zeng, M., Nguyen, L.T., Yu, B., Mengshoel, O.J., Zhu, J., Wu, P., Zhang, J.: Convolutional neural networks for human activity recognition using mobile sensors. In: MobiCASE, pp. 197–205. IEEE (2014)

    Google Scholar 

  28. LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. In: Procedings of the IEEE, pp. 2278–2324, November 1998

    Article  Google Scholar 

  29. Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: Computer Vision and Pattern Recognition (CVPR09) (2009)

    Google Scholar 

  30. Zheng, Y., Liu, Q., Chen, E., Ge, Y., Zhao, J.L.: Time series classification using multi-channels deep convolutional neural networks. In: Proceedings of International Conference on Web-Age Information Management, pp. 298–310 (2014)

    Google Scholar 

  31. Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning (Adaptive Computation and Machine Learning series). MIT Press, Chap. 10, pp. 330–372 (2016)

    Google Scholar 

  32. Ronao, C.A., Cho, S.-B.: Deep convolutional neural networks for human activity recognition with smartphone sensors. In: Neural Information Processing, pp. 46–53. Springer (2015)

    Google Scholar 

  33. Ronao, C.A., Cho, S.-B.: Evaluation of deep convolutional neural network architectures for human activity recognition with smartphone sensors. In: Proceedings of the KIISE Korea Computer Congress, pp. 858–860 (2015)

    Google Scholar 

  34. Yang, J.B., Nguyen, M.N., San, P., Li, X., Krishnaswamy, S.: Deep convolutional neural networks on multichannel time series for human activity recognition. In: IJCAI 2015 Proceedings of the 24th International Conference on Artificial Intelligence, pp. 3995–4001 (2015)

    Google Scholar 

  35. Alsheikh, M.A., Selim, A., Niyato, D., Doyle, L., Lin, S., Tan, H.-P.: Deep Activity Recognition Models with Triaxial Accelerometers. In: The Workshops of the Thirtieth AAAI Conference on Artificial Intelligence (2015)

    Google Scholar 

  36. Jiang, W., Yin, Z.: Human activity recognition using wearable sensors by deep convolutional neural networks. In: Proceedings of the 23rd ACM international conference on Multimedia, pp. 1307–1310 (2015)

    Google Scholar 

  37. Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning internal representations by error propagation. In: Parallel Distributed Processing, vol. 1, Chap. 8, pp. 318–362. MIT Press, Cambridge (1986)

    Google Scholar 

  38. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)

    Article  Google Scholar 

  39. Ordóñez, F.J., Roggen, D.: Deep convolutional feature transfer across mobile activity recognition domains, sensor modalities and locations. In: Proceedings of IEEE 20th International Symposium on Wearable Computers (ISWC), pp. 92–99 (2016)

    Google Scholar 

  40. Luo, W., Li, Y., Urtasun, R., Zemel, R.: Understanding the effective receptive field in deep convolutional neural networks. In: 29th Conference on Neural Information Processing Systems (NIPS) (2016)

    Google Scholar 

  41. Stisen, A., Blunck, H., Bhattacharya, S., Prentow, T.S., Kjærgaard, M.B., Dey, A., Sonne, T., Jensen, M.M.: Smart devices are different: assessing and mitigating mobile sensing heterogeneities for activity recognition. In: Proceedings of the 13th ACM Conference on Embedded Networked Sensor Systems, pp. 127–140, November 2015

    Google Scholar 

  42. Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15, 1929–1958 (2014)

    MathSciNet  MATH  Google Scholar 

  43. Lin, M., Chen, Q., Yan, S.: Network in network. In: International Conference on Learning Representations (2014)

    Google Scholar 

  44. Zeiler, M.D.: ADADELTA: An Adaptive Learning Rate Method. Technical report, arXiv 1212.5701

    Google Scholar 

  45. Anguita, D., Ghio, A., Oneto, L., Parra, X., Reyes-Ortiz, J.L.: A public domain dataset for human activity recognition using smartphones. In: 21th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning, ESANN 2013, April 2013

    Google Scholar 

  46. Reiss, A., Stricker, D.: Introducing a new benchmarked dataset for activity monitoring. In: The 16th IEEE International Symposium on Wearable Computers (ISWC) (2012)

    Google Scholar 

  47. Reiss, A., Weber, M., Stricker, D.: Exploring and extending the boundaries of physical activity recognition. In: IEEE SMC Workshop on Robust Machine Learning Techniques for Human Activity Recognition, pp. 46–50 (2011)

    Google Scholar 

  48. Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the 13th International Conference on Artificial Intelligence and Statistics (AISTATS), pp. 249–256 (2010)

    Google Scholar 

  49. He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: surpassing human-level performance on ImageNet classification. In: IEEE International Conference on Computer Vision (ICCV) (2015)

    Google Scholar 

  50. van der Maaten, L.J.P., Hinton, G.E.: Visualizing high-dimensional data using t-SNE. J. Mach. Learn. Res. 9, 2579–2605 (2008)

    MATH  Google Scholar 

  51. Ermes, M., Pärkkä, J., Mäntyjärvi, J., Korhonen, I.: Detection of daily activities and sports with wearable sensors in controlled and uncontrolled conditions. IEEE Trans. Inf. Technol. Biomed. 12(1), 20–26 (2008)

    Article  Google Scholar 

  52. Reiss, A., Stricker, D.: Creating and benchmarking a new dataset for physical activity monitoring. In: Proceedings of the 5th International Conference on PErvasive Technologies Related to Assistive Environments, Article no. 40, June 2012

    Google Scholar 

  53. Sabour, S., Frosst, N., Hinton, G.E.: Dynamic routing between capsules. In: 31st Conference on Neural Information Processing Systems (NIPS) (2017)

    Google Scholar 

Download references

Acknowledgment

This work is funded by the European Commission under project TRILLION, grant number H2020-FCT-2014, REA grant agreement no [653256]. Moreover, we gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan-X GPU used for this research.

Author information

Authors and Affiliations

  1. Department of Electrical and Electronics Engineering, University of West Attica, Athens, Greece

    Panagiotis Kasnesis & Charalampos Z. Patrikakis

  2. School of Electrical and Computer Engineering, National Technical University of Athens, Athens, Greece

    Iakovos S. Venieris

Authors
  1. Panagiotis Kasnesis
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Charalampos Z. Patrikakis
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Iakovos S. Venieris
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Panagiotis Kasnesis .

Editor information

Editors and Affiliations

  1. Faculty of Science and Engineering, Saga University, Saga, Japan

    Kohei Arai

  2. The Science and Information (SAI) Organization, Bradford, UK

    Supriya Kapoor

  3. The Science and Information (SAI) Organization, Bradford, UK

    Rahul Bhatia

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Kasnesis, P., Patrikakis, C.Z., Venieris, I.S. (2019). PerceptionNet: A Deep Convolutional Neural Network for Late Sensor Fusion. In: Arai, K., Kapoor, S., Bhatia, R. (eds) Intelligent Systems and Applications. IntelliSys 2018. Advances in Intelligent Systems and Computing, vol 868. Springer, Cham. https://doi.org/10.1007/978-3-030-01054-6_7

Download citation

Publish with us

Policies and ethics

Search

Navigation