Abstract
In this paper, we present an algorithm for action recognition that uses only depth maps. At the beginning we extract features describing the person shape in single depth maps. For each class we train a separate one-against-all convolutional neural network to extract class-specific features. The actions are represented by multivariate time-series of such CNN-based frame features for which we calculate statistical features. For the non-zero pixels representing the person shape in each depth map we calculate handcrafted features. For time-series of such handcrafted features we calculate the statistical features. Afterwards, handcrafted features that are common for all actions and CNN-based features that are action-specific are concatenated together resulting in action feature vectors. For each action feature vector we train a multi-class classifier with one-hot encoding of output labels. The prediction of the action is done by a voting-based ensemble operating on such one-hot encoding outputs. We demonstrate experimentally that on UTD-MHAD dataset the proposed algorithm outperforms state-of-the-art depth-based algorithms and achieves promising results on MSR-Action3D dataset.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Aggarwal, J., Ryoo, M.: Human activity analysis: a review. ACM Comput. Surv. 43(3), 16:1–16:43 (2011)
Liang, B., Zheng, L.: A survey on human action recognition using depth sensors. In: International Conference on Digital Image Computing: Techniques and Applications, pp. 1–8 (2015)
Aggarwal, J., Xia, L.: Human activity recognition from 3D data: a review. Pattern Recognit. Lett. 48, 70–80 (2014)
Chen, L., Wei, H., Ferryman, J.: A survey of human motion analysis using depth imagery. Pattern Recognit. Lett. 34(15), 1995–2006 (2013)
Ye, M., Zhang, Q., Wang, L., Zhu, J., Yang, R., Gall, J.: A survey on human motion analysis from depth data. In: Grzegorzek, M., Theobalt, C., Koch, R., Kolb, A. (eds.) Time-of-Flight and Depth Imaging. Sensors, Algorithms, and Applications. LNCS, vol. 8200, pp. 149–187. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-44964-2_8
Lo Presti, L., La Cascia, M.: 3D skeleton-based human action classification. Pattern Recognit. 53(C), 130–147 (2016)
Xia, L., Chen, C.C., Aggarwal, J.: View invariant human action recognition using histograms of 3D joints. In: CVPR Workshops, pp. 20–27 (2012)
Li, W., Zhang, Z., Liu, Z.: Action recognition based on a bag of 3D points. In: IEEE International Conference on Computer Vision and Pattern Recognition - Workshops, pp. 9–14 (2010)
Chen, C., Jafari, R., Kehtarnavaz, N.: Action recognition from depth sequences using depth motion maps-based local binary patterns. In: 2015 IEEE Winter Conference on Applications of Computer Vision, pp. 1092–1099 (2015)
Yang, X., Zhang, C., Tian, Y.L.: Recognizing actions using depth motion maps-based histograms of oriented gradients. In: Proceedings of the 20th ACM International Conference on Multimedia, pp. 1057–1060. ACM (2012)
Wang, J., Liu, Z., Chorowski, J., Chen, Z., Wu, Y.: Robust 3D action recognition with random occupancy patterns. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, pp. 872–885. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33709-3_62
Vieira, A.W., Nascimento, E.R., Oliveira, G.L., Liu, Z., Campos, M.F.M.: STOP: space-time occupancy patterns for 3D action recognition from depth map sequences. In: Alvarez, L., Mejail, M., Gomez, L., Jacobo, J. (eds.) CIARP 2012. LNCS, vol. 7441, pp. 252–259. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33275-3_31
Xia, L., Aggarwal, J.: Spatio-temporal depth cuboid similarity feature for activity recognition using depth camera. In: IEEE International Conference on Computer Vision and Pattern Recognition, pp. 2834–2841 (2013)
Oreifej, O., Liu, Z.: HON4D: histogram of oriented 4D normals for activity recognition from depth sequences. In: IEEE International Conference on Computer Vision and Pattern Recognition, pp. 716–723 (2013)
Wang, P., Li, W., Gao, Z., Zhang, J., Tang, C., Ogunbona, P.: Action recognition from depth maps using deep convolutional neural networks. IEEE Trans. Hum.-Mach. Syst. 46(4), 498–509 (2016)
Chen, C., Jafari, R., Kehtarnavaz, N.: UTD-MHAD: a multimodal dataset for human action recognition utilizing a depth camera and a wearable inertial sensor. In: 2015 IEEE International Conference on Image Processing (ICIP), pp. 168–172, September 2015
Wang, P., Li, W., Li, C., Hou, Y.: Action recognition based on joint trajectory maps with convolutional neural networks. Knowl.-Based Syst. 158, 43–53 (2018)
Hou, Y., Li, Z., Wang, P., Li, W.: Skeleton optical spectra-based action recognition using convolutional neural networks. IEEE Trans. Circuits Syst. Video Technol. 28(3), 807–811 (2018)
Hussein, M.E., Torki, M., Gowayyed, M.A., El-Saban, M.: Human action recognition using a temporal hierarchy of covariance descriptors on 3D joint locations. In: Proceedings of the Twenty-Third International Joint Conferences on Artificial Intelligence, IJCAI 2013, pp. 2466–2472. AAAI Press (2013)
Zhou, L., Li, W., Zhang, Y., Ogunbona, P., Nguyen, D., Zhang, H.: Discriminative key pose extraction using extended LC-KSVD for action recognition. In: 2014 International Conference on Digital Image Computing: Techniques and Applications (DICTA), pp. 1–8 (2014)
Wang, P., Wang, S., Gao, Z., Hou, Y., Li, W.: Structured images for RGB-D action recognition. In: 2017 IEEE International Conference on Computer Vision Workshops (ICCVW), pp. 1005–1014 (2017)
Wu, Y.: Mining actionlet ensemble for action recognition with depth cameras. In: IEEE International Conference on Computer Vision and Pattern Recognition, pp. 1290–1297 (2012)
Wang, P., Li, W., Gao, Z., Zhang, J., Tang, C., Ogunbona, P.O.: Action recognition from depth maps using deep convolutional neural networks. IEEE Trans. Hum.-Mach. Syst. 46(4), 498–509 (2016)
Lu, C., Jia, J., Tang, C.: Range-sample depth feature for action recognition. In: IEEE International Conference on Computer Vision and Pattern Recognition, pp. 772–779 (2014)
Acknowledgment
This work was supported by Polish National Science Center (NCN) under a research grant 2017/27/B/ST6/01743.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Trelinski, J., Kwolek, B. (2019). Ensemble of Classifiers Using CNN and Hand-Crafted Features for Depth-Based Action Recognition. In: Rutkowski, L., Scherer, R., Korytkowski, M., Pedrycz, W., Tadeusiewicz, R., Zurada, J. (eds) Artificial Intelligence and Soft Computing. ICAISC 2019. Lecture Notes in Computer Science(), vol 11509. Springer, Cham. https://doi.org/10.1007/978-3-030-20915-5_9
Download citation
DOI: https://doi.org/10.1007/978-3-030-20915-5_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-20914-8
Online ISBN: 978-3-030-20915-5
eBook Packages: Computer ScienceComputer Science (R0)