Abstract
Most concept recognition in visual multimedia is based on relatively simple concepts, things which are present in the image or video. These usually correspond to objects which can be identified in images or individual frames. Yet there is also a need to recognise semantic concepts which have a temporal aspect corresponding to activities or complex events. These require some form of time series for recognition and also require some individual concepts to be detected so as to utilise their time-varying features, such as co-occurrence and re-occurrence patterns. While results are reported in the literature of using concept detections which are relatively specific and static, there are research questions which remain unanswered. What concept detection accuracies are satisfactory for time series recognition? Can recognition methods perform equally well across various concept detection performances? What affecting factors need to be taken into account when building concept-based high-level event/activity recognitions? In this paper, we conducted experiments to investigate these questions. Results show that though improving concept detection accuracies can enhance the recognition of time series based concepts, they do not need to be very accurate in order to characterize the dynamic evolution of time series if appropriate methods are used. Experimental results also point out the importance of concept selection for time series recognition, which is usually ignored in the current literature.
P. Wang—Work was part-funded by 973 Program under Grant No. 2011CB302206, National Natural Science Foundation of China under Grant No. 61272231, 61472204, 61502264, Beijing Key Laboratory of Networked Multimedia.
A.F. Smeaton—Supported by Science Foundation Ireland under grant number SFI/12/RC/2289.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Gupta, A., Srinivasan, P., Shi, J., Davis, L.: Understanding videos, constructing plots learning a visually grounded storyline model from annotated videos. In: CVPR 2009, pp. 2012–2019, June 2009
Pirsiavash, H., Ramanan, D.: Detecting activities of daily living in first-person camera views. In: CVPR 2012, pp. 2847–2854, June 2012
Tan, C.C., Jiang, Y.-G., Ngo, C.-W.: Towards textually describing complex video contents with audio-visual concept classifiers. In: ACM Multimedia 2011, pp. 655–658 (2011)
Yu, Q., Liu, J., Cheng, H., Divakaran, A., Sawhney, H.: Multimedia event recounting with concept based representation. In: ACM Multimedia 2012, pp. 1073–1076 (2012)
Liu, J., Yu, Q., Javed, O., Ali, S., Tamrakar, A., Divakaran, A., Cheng, H., Sawhney, H.: Video event recognition using concept attributes. In: IEEE Winter Conference on Applications of Computer Vision, pp. 339–346 (2013)
Merler, M., Huang, B., Xie, L., Hua, G., Natsev, A.: Semantic model vectors for complex video event recognition. IEEE Trans. Multimedia 14(1), 88–101 (2012)
Bhattacharya, S., Kalayeh, M., Sukthankar, R., Shah, M.: Recognition of complex events exploiting temporal dynamics between underlying concepts. In: CVPR 2014, June 2014
Hill, M., Hua, G., Huang, B., Merler, M., Natsev, A., Smith, J.R., Xie, L., Ouyang, H., Zhou, M.: IBM research TRECVid-2010 video copy detection and multimedia event detection system (2010)
Cheng, H., Liu, J., Ali, S., Javed, O., Yu, Q., Tamrakar, A., Divakaran, A., Sawhney, H.S., Manmatha, R., Allan, J., Hauptmann, A., Shah, M., Bhattacharya, S., Dehghan, A., Friedland, G., Elizalde, B.M., Darrell, T., Witbrock, M., Curtis, J.: SRI-Sarnoff AURORA system at TRECVid 2012: Multimedia event detection and recounting. In: NIST TRECVID Workshop, December 2012
Doherty, A.R., Caprani, N., O’Conaire, C., Kalnikaite, V., Gurrin, C., O’Connor, N.E., Smeaton, A.F.: Passively recognising human activities through lifelogging. Comput. Hum. Behav. 27(5), 1948–1958 (2011)
Wang, P., Smeaton, A.F.: Using visual lifelogs to automatically characterise everyday activities. Inf. Sci. 230, 147–161 (2013)
Wang, P., Smeaton, A.: Semantics-based selection of everyday concepts in visual lifelogging. Int. J. Multimedia Inf. Retrieval 1(2), 87–101 (2012)
Aly, R., Hiemstra, D., de Jong, F., Apers, P.: Simulating the future of concept-based video retrieval under improved detector performance. Multimedia Tools Appl. 60, 1–29 (2011)
Guo, J., Scott, D., Hopfgartner, F., Gurrin, C.: Detecting complex events in user-generated video using concept classifiers. In: CBMI 2012, pp. 1–6, June 2012
Laptev, I., Marszalek, M., Schmid, C., Rozenfeld, B.: Learning realistic human actions from movies. In: CVPR 2008, pp. 1–8, June 2008
Jaakkola, T.S., Haussler, D.: Exploiting generative models in discriminative classifiers. In: NIPS 1999, pp. 487–493 (1999)
Sun, C., Nevatia, R.: ACTIVE: activity concept transitions in video event classification. In: ICCV 2013, pp. 913–920, December 2013
Li, W., Yu, Q., Sawhney, H., Vasconcelos, N.: Recognizing activities via bag of words for attribute dynamics. In: CVPR 2013, pp. 2587–2594, June 2013
Liu, J., Yu, Q., Javed, O., Ali, S., Tamrakar, A., Divakaran, A., Cheng, H., Sawhney, H.: Video event recognition using concept attributes. In: WACV 2013, pp. 339–346 (2013)
van der Maaten, L.: Learning discriminative fisher kernels. In: ICML 2011, pp. 217–224 (2011)
Assari, S., Zamir, A., Shah, M.: Video classification using semantic concept co-occurrences. In: CVPR 2014, pp. 2529–2536, June 2014
Wang, P., Smeaton, A.F., Gurrin, C.: Factorizing time-aware multi-way tensors for enhancing semantic wearable sensing. In: He, X., Luo, S., Tao, D., Xu, C., Yang, J., Hasan, M.A. (eds.) MMM 2015, Part I. LNCS, vol. 8935, pp. 571–582. Springer, Heidelberg (2015)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing Switzerland
About this paper
Cite this paper
Wang, P., Sun, L., Yang, S., Smeaton, A.F. (2016). What are the Limits to Time Series Based Recognition of Semantic Concepts?. In: Tian, Q., Sebe, N., Qi, GJ., Huet, B., Hong, R., Liu, X. (eds) MultiMedia Modeling. MMM 2016. Lecture Notes in Computer Science(), vol 9517. Springer, Cham. https://doi.org/10.1007/978-3-319-27674-8_25
Download citation
DOI: https://doi.org/10.1007/978-3-319-27674-8_25
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-27673-1
Online ISBN: 978-3-319-27674-8
eBook Packages: Computer ScienceComputer Science (R0)