Advertisement

A Survey on Human Group Activity Recognition by Analysing Person Action from Video Sequences Using Machine Learning Techniques

  • Smita Kulkarni
  • Sangeeta Jadhav
  • Debashis AdhikariEmail author
Chapter
Part of the Algorithms for Intelligent Systems book series (AIS)

Abstract

There has been a tremendous advance in machine learning techniques especially for automatic group activity recognition (GAR) over the past few decades. This review article surveys the modern advancement made towards video-based group activity recognition technique. Various applications, including video surveillance systems, sports analytics and human behaviour for robotics characterization, require a group activity recognition system. Comprehensive reviews of machine learning (ML) techniques like hidden Markov models (HMMs), graphical method and support vector machines employed in GAR are being discussed. A comprehensive review on the latest progress in deep learning model has delivered important developments in GAR performance; those are also presented. The main purpose of this survey is to broadly categorize and analyse GAR according to handcrafted features based on machine learning model and learned features based on deep model. Various GAR models illustrated by considering activities of individual person, person-to-person interaction, person-to-group interaction and group interaction using temporal sequence information from video frames for recognition of group activity are discussed. The review facilitates in diverse applications, and the models described in different applications present specifically in surveillance, sport analytics, video summary, etc.

References

  1. 1.
    Blank M, Gorelick L, Shechtman E, Irani M, Basri R (2005) Actions as space-time shapes. In: ICCVGoogle Scholar
  2. 2.
    Schuldt C, Laptev I, Caputo B (2004) Recognizing human actions: a local SVM approach. In: ICPRGoogle Scholar
  3. 3.
    Wang H, Schmid C (2013) Action recognition with improved trajectories. In: Proceedings of the IEEE international conference on computer vision, pp 3551–3558Google Scholar
  4. 4.
    Turaga P, Chellappa R, Subrahmanian VS, Udrea O (2008) Machine recognition of human activities: a survey. IEEE Trans Circuits Syst Video Technol 18(11):1473CrossRefGoogle Scholar
  5. 5.
    Aggarwal JK, Ryoo (2011) MS human activity analysis. ACM Comput Surv 43(3):1–43CrossRefGoogle Scholar
  6. 6.
    Ke S-R, Thuc H, Lee Y-J, Hwang J-N, Yoo J-H, Choi K-H (2013) A review on video-based human activity recognition. Computers 2(2):88–131CrossRefGoogle Scholar
  7. 7.
    Vahora SA, Chauhan NC (2017) A comprehensive study of group activity recognition methods in video. Indian J Sci Technol 10(23):1–11CrossRefGoogle Scholar
  8. 8.
    Stergiou A, Poppe R (2018) Understanding human-human interactions: a survey. arXiv:1808.00022
  9. 9.
    Vaswani N, Roy Chowdhury A, Chellappa R (2003) Activity recognition using the dynamics of the configuration of interacting objects. In: 2003 IEEE computer society conference on computer vision and pattern recognition, proceedings, vol 2, pp II-633Google Scholar
  10. 10.
    Khan SM, Shah M (2005) Detecting group activities using rigidity of formation. In: Proceedings of the 13th annual ACM international conference on multimedia, pp 403–406Google Scholar
  11. 11.
    Intille SS, Bobick AF (2001) Recognizing planned multiperson action. Comput Vis Image Underst 81(3):414–445CrossRefGoogle Scholar
  12. 12.
    Moore D, Essa I (2002) Recognizing multitasked activities from video using stochastic context-free grammar. In: AAAI/IAAI, pp 770–776Google Scholar
  13. 13.
    Cupillard F, Brémond F, Thonnat M (2002) Group behavior recognition with multiple cameras. In: Sixth IEEE workshop on applications of computer vision, proceedings, pp 177–183Google Scholar
  14. 14.
    Chang M-C, Krahnstoever N, Lim S, Yu T (2010) Group level activity recognition in crowded environments across multiple cameras. In: 7th IEEE international conference on advanced video and signal based surveillance, pp 56–63Google Scholar
  15. 15.
    Ryoo MS, Aggarwal JK (2009) Stochastic representation and recognition of high-level group activities: describing structural uncertainties in human activities. In: 2009 IEEE computer society conference on computer vision and pattern recognition workshops, pp 11–11Google Scholar
  16. 16.
    Ryoo MS, Aggarwal JK (2011) Stochastic representation and recognition of high-level group activities. Int J Comput Vision 93(2):183–200MathSciNetCrossRefGoogle Scholar
  17. 17.
    Zhang D, Gatica-Perez D, Bengio S, McCowan I (2006) Modeling individual and group actions in meetings with layered HMMs. IEEE Trans Multimedia 8(3):509–520CrossRefGoogle Scholar
  18. 18.
    Lin W, Sun M-T, Poovendran R, Zhang Z (2010) Group event detection with a varying number of group members for video surveillance. IEEE Trans Circuits Syst Video Technol 20(8):1057–1067Google Scholar
  19. 19.
    Zaidenberg S, Boulay B, Brémond F (2012) A generic framework for video understanding applied to group behavior recognition. In: IEEE ninth international conference on advanced video and signal-based surveillance. Beijing, pp 136–142Google Scholar
  20. 20.
    Guo P, Miao Z, Zhang X, Shen Y, Wang S (2012) Coupled observation decomposed hidden markov model for multiperson activity recognition. IEEE Trans Circuits Syst Video Technol 22(9):1306–1320CrossRefGoogle Scholar
  21. 21.
    Lin W, Chu H, Wu J, Sheng B, Chen Z (2013) A heat-map-based algorithm for recognizing group activities in videos. IEEE Trans Circuits Syst Video Technol 23(11):1980–1992CrossRefGoogle Scholar
  22. 22.
    Choi W, Shahid K, Savarese S (2009) What are they doing? Collective activity classification using spatio-temporal relationship among people. In: IEEE 12th international conference on computer vision workshops, ICCV workshops, pp 1282–1289Google Scholar
  23. 23.
    Gupta A, Srinivasan P, Shi J, Davis LS (2009) Understanding videos, constructing plots learning a visually grounded storyline model from annotated videos. In: IEEE conference on computer vision and pattern recognition, pp 2012–2019Google Scholar
  24. 24.
    Amer MR, Xie D, Zhao M, Todorovic S, Zhu S-C (2012) Cost-sensitive top-down/bottom-up inference for multiscale activity recognition. European conference on computer vision. Springer, Berlin, Heidelberg, pp 187–200Google Scholar
  25. 25.
    Amer MR, Todorovic S, Fern A, Zhu S-C (2013) Monte carlo tree search for scheduling activity recognition. In: Proceedings of the IEEE international conference on computer vision, pp 1353–1360Google Scholar
  26. 26.
    Lan T, Wang Y, Yang W, Mori G (2010) Beyond actions: discriminative models for contextual group activities. In: Advances in neural information processing systems, pp 1216–1224Google Scholar
  27. 27.
    Lan T, Wang Y, Yang W, Robinovitch SN, Mori G (2012) Discriminative latent models for recognizing contextual group activities. IEEE Trans Pattern Anal Mach Intell 34(8):1549–1562CrossRefGoogle Scholar
  28. 28.
    Lan T, Sigal L, Mori G (2012) Social roles in hierarchical models for human activity recognition. In: IEEE conference on computer vision and pattern recognition, pp 1354–1361Google Scholar
  29. 29.
    Choi W, Savarese S (2012) A unified framework for multi-target tracking and collective activity recognition. In: European conference on computer vision. Springer, Berlin, Heidelberg, pp 215–230Google Scholar
  30. 30.
    Amer MR, Lei P, Todorovic S (2014) Hirf: hierarchical random field for collective activity recognition in videos. In: European conference on computer vision. Springer, pp 572–585Google Scholar
  31. 31.
    Choi W, Chao YW, Pantofaru C, Savarese S (2012) Discovering groups of people in images. In: European conference on computer vision. Springer, pp 417–433Google Scholar
  32. 32.
    Khamis S, Morariu VI, Davis LS (2012) Combining per-frame and per-track cues for multi-person action recognition. European conference on computer vision. Springer, Berlin, Heidelberg, pp 116–129Google Scholar
  33. 33.
    Hajimirsadeghi H, Mori G (2015) Learning ensembles of potential functions for structured prediction with latent variables. In: Proceedings of the IEEE international conference on computer vision, pp 4059–4067Google Scholar
  34. 34.
    Zhu Y, Nayak NM, Roy-Chowdhury AK (2013) Contextaware modeling and recognition of activities in video. In: Computer vision and pattern recognition (CVPR), IEEE conference, pp 2491–2498Google Scholar
  35. 35.
    Tran KN, Gala A, Kakadiaris IA, Shah SK (2014) Activity analysis in crowded environments using social cues for group discovery and human interaction modeling. Pattern Recogn Lett 44:49–57CrossRefGoogle Scholar
  36. 36.
    Deng Z, Zhai M, Chen L, Liu Y, Muralidharan S, Roshtkhari MJ, Mori G (2015) Deep structured models for group activity recognition. In: British machine vision conference, pp 179.1–179Google Scholar
  37. 37.
    Hajimirsadeghi H, Yan W, Vahdat A, Mori G (2015) Visual recognition by counting instances: a multi-instance cardinality potential kernel. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2596–2605Google Scholar
  38. 38.
    Deng Z, Vahdat A, Hu H, Mori G (2016) Structure inference machines: recurrent neural networks for analyzing relations in group activity recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4772–4781Google Scholar
  39. 39.
    Li W, Chang MC, Lyu S (2018) Who did what at where and when: simultaneous multi-person tracking and activity recognition. arXiv:1807.01253
  40. 40.
    Kaneko T, Shimosaka M, Odashima S, Fukui R, Sato T (2014) A fully connected model for consistent collective activity recognition in videos. Pattern Recogn Lett 43:109–118CrossRefGoogle Scholar
  41. 41.
    Hasan M, Roy-Chowdhury AK (2015) A continuous learning framework for activity recognition using deep hybrid feature models. IEEE Trans Multimedia 17(11):1909–1922CrossRefGoogle Scholar
  42. 42.
    Bisagno N, Zhang B, Conci N (2018) Group LSTM: group trajectory prediction in crowded scenarios. In: Proceedings of the European conference on computer vision (ECCV)Google Scholar
  43. 43.
    Karpathy A, Toderici G, Shetty S, Leung T, Sukthankar R, Fei-Fei L (2014) Large-scale video classification with convolutional neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1725–1732Google Scholar
  44. 44.
    Yeung S, Russakovsky O, Mori G, Fei-Fei L (2016) End-to-end learning of action detection from frame glimpses in videos. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2678–2687Google Scholar
  45. 45.
    Ramanathan V, Huang J, Abu-El-Haija S, Gorban A, Murphy K, Fei-Fei L (2016) Detecting events and key actors in multi-person videos. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3043–3053Google Scholar
  46. 46.
    Ibrahim MS, Muralidharan S, Deng Z, Vahdat A, Mori G (2016) A hierarchical deep temporal model for group activity recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1971–1980Google Scholar
  47. 47.
    Ibrahim MS, Muralidharan S, Deng Z, Vahdat A, Mori G (2016) Hierarchical deep temporal models for group activity recognition. arXiv:1607.02643
  48. 48.
    Tora MR, Chen J, Little JJ (2017) Classification of puck possession events in ice hockey. In: 2017 IEEE conference on computer vision and pattern recognition workshops (CVPRW), pp 147–154Google Scholar
  49. 49.
    Ibrahim MS, Mori G (2018) Hierarchical relational networks for group activity recognition and retrieval. In: Proceedings of the European conference on computer vision (ECCV), pp 721–736Google Scholar
  50. 50.
    Shu T, Todorovic S, Zhu S-C (2017) CERN: confidence-energy recurrent network for group activity recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5523–5531Google Scholar
  51. 51.
    Li X, Chuah MC (2017) Sbgar: semantics based group activity recognition. In: Proceedings of the IEEE international conference on computer vision, pp 2876–2885Google Scholar
  52. 52.
    Shu X, Tang J, Qi G-J, Liu W, Yang J (2018) Hierarchical long short-term concurrent memory for human interaction recognition. arXiv:1811.00270
  53. 53.
    Tsunoda T, Komori Y, Matsugu M, Harada T (2017) Football action recognition using hierarchical LSTM. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 99–107Google Scholar
  54. 54.
    Yan R, Tang J, Shu X, Li Z, Tian Q (2018) Participation-contributed temporal dynamic model for group activity recognition. In: ACM multimedia conference on multimedia conference, pp 1292–1300Google Scholar
  55. 55.
    Azar SM, Atigh MG, Nickabadi A (2018) A multi-stream convolutional neural network framework for group activity recognition. arXiv:1812.10328
  56. 56.
    Bagautdinov T, Alahi A, Fleuret F, Fua P, Savarese S (2017) Social scene understanding: end-to-end multi-person action localization and collective activity recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4315–4324Google Scholar
  57. 57.
    Biswas S, Gall J (2018) Structural recurrent neural network (SRNN) for group activity analysis. In: IEEE winter conference on applications of computer vision (WACV), pp 1625–1632Google Scholar
  58. 58.
    Tang Y, Wang Z, Li P, Lu J, Yang M, Zhou J (2018) Mining semantics-preserving attention for group activity recognition. In: 2018 ACM multimedia conference on multimedia conference, pp 1283–1291Google Scholar
  59. 59.
    Lu L, Di H, Yao L, Zhang L, Wang S (2019) Spatio-temporal attention mechanisms based model for collective activity recognition. Sig Process Image Commun 74:162–174CrossRefGoogle Scholar
  60. 60.
    Vahora SA, Chauhan NC (2019) Deep neural network model for group activity recognition using contextual relationship. Eng Sci Technol Int J 22(1):47–54CrossRefGoogle Scholar
  61. 61.
    Qi M, Wang Y, Qin J, Li A, Luo J, Van Gool L (2019) StagNet an attentive semantic RNN for group activity and individual action recognition. IEEE Trans Circuits Syst Video TechnolGoogle Scholar
  62. 62.
    Gammulle H, Denman S, Sridharan S, Fookes C (2018) Multi-level sequence GAN for group activity recognition. arXiv:1812.07124

Copyright information

© Springer Nature Singapore Pte Ltd. 2020

Authors and Affiliations

  • Smita Kulkarni
    • 1
    • 3
  • Sangeeta Jadhav
    • 2
  • Debashis Adhikari
    • 3
    Email author
  1. 1.D.Y. Patil College of EngineeringPuneIndia
  2. 2.Army Institute of TechnologyPuneIndia
  3. 3.MIT Academy of EngineeringPuneIndia

Personalised recommendations