A Deep Learning Approach to Predict Crowd Behavior Based on Emotion

  • Elizabeth B. Varghese
  • Sabu M. ThampiEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11010)


In a visual surveillance system, predicting crowd behavior has recently emerged as a crucial problem for crowd management and monitoring. Specifically, potential dangers and disasters can be avoided by correctly detecting crowd behavior. In this paper, we propose an approach to forecast crowd behavior using a deep learning framework and multiclass Support Vector Machine (SVM). We extract spatio-temporal descriptors using 3D Convolutional Neural Network (3DCNN) based on crowd emotions. In particular, the learned emotion based descriptors help to build the semantic ambiguity in classifying crowd behavior. The effectiveness of our approach is validated with 3 benchmark datasets: Motion Emotion Dataset (MED), ViolentFlows and UMN. The obtained results prove that our approach is successful in predicting crowd behavior in challenging situations. Our system also outperforms existing methods that use local feature descriptors, which reveals that emotions from spatio-temporal features are beneficial for the correct anticipation of crowd behavior.


Crowd emotion Crowd behavior Spatio-temporal features 3D Convolutional Neural Network (3DCNN) Multiclass Support Vector Machine (SVM) 


  1. 1.
    Detection of Unusual Crowd Activity. Accessed 7 Apr 2018
  2. 2.
    Baig, M.W., Barakova, E.I., Marcenaro, L., Rauterberg, M., Regazzoni, C.S.: Crowd emotion detection using dynamic probabilistic models. In: del Pobil, A.P., Chinellato, E., Martinez-Martin, E., Hallam, J., Cervera, E., Morales, A. (eds.) SAB 2014. LNCS, vol. 8575, pp. 328–337. Springer, Cham (2014). Scholar
  3. 3.
    Blondel, M., Fujino, A., Ueda, N.: Large-scale multiclass support vector machine training via euclidean projection onto the simplex. In: 2014 22nd International Conference on Pattern Recognition, ICPR, pp. 1289–1294. IEEE (2014)Google Scholar
  4. 4.
    Dupont, C., Tobías, L., Luvison, B.: Crowd-11: a dataset for fine grained crowd behaviour analysis. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 9–16 (2017)Google Scholar
  5. 5.
    Guy, S.J., Kim, S., Lin, M.C., Manocha, D.: Simulating heterogeneous crowd behaviors using personality trait theory. In: Proceedings of the 2011 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, pp. 43–52. ACM (2011)Google Scholar
  6. 6.
    Hassner, T., Itcher, Y., Kliper-Gross, O.: Violent flows: real-time detection of violent crowd behavior. In: 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, CVPRW, pp. 1–6. IEEE (2012)Google Scholar
  7. 7.
    Laptev, I., Marszalek, M., Schmid, C., Rozenfeld, B.: Learning realistic human actions from movies. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2008, pp. 1–8. IEEE (2008)Google Scholar
  8. 8.
    Mehran, R., Oyama, A., Shah, M.: Abnormal crowd behavior detection using social force model. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2009, pp. 935–942. IEEE (2009)Google Scholar
  9. 9.
    Mousavi, H., Mohammadi, S., Perina, A., Chellali, R., Murino, V.: Analyzing tracklets for the detection of abnormal crowd behavior. In: 2015 IEEE Winter Conference on Applications of Computer Vision, WACV, pp. 148–155. IEEE (2015)Google Scholar
  10. 10.
    Rabiee, H., Haddadnia, J., Mousavi, H.: Crowd behavior representation: an attribute-based approach. SpringerPlus 5(1), 1179 (2016)CrossRefGoogle Scholar
  11. 11.
    Rabiee, H., Haddadnia, J., Mousavi, H., Kalantarzadeh, M., Nabi, M., Murino, V.: Novel dataset for fine-grained abnormal behavior understanding in crowd. In: 2016 13th IEEE International Conference on Advanced Video and Signal Based Surveillance, AVSS, pp. 95–101. IEEE (2016)Google Scholar
  12. 12.
    Sabokrou, M., Fayyaz, M., Fathy, M., et al.: Fully convolutional neural network for fast anomaly detection in crowded scenes. arXiv preprint arXiv:1609.00866 (2016)
  13. 13.
    Saifi, L., Boubetra, A., Nouioua, F.: An approach for emotions and behavior modeling in a crowd in the presence of rare events. Adapt. Behav. 24(6), 428–445 (2016)CrossRefGoogle Scholar
  14. 14.
    Shao, J., Kang, K., Change Loy, C., Wang, X.: Deeply learned attributes for crowded scene understanding. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4657–4666 (2015)Google Scholar
  15. 15.
    Amir Sjarif, N.N., Shamsuddin, S.M., Mohd Hashim, S.Z., Yuhaniz, S.S.: Crowd analysis and its applications. In: Mohamad Zain, J., Wan Mohd, W.M., El-Qawasmeh, E. (eds.) ICSECS 2011. CCIS, vol. 179, pp. 687–697. Springer, Heidelberg (2011). Scholar
  16. 16.
    Tran, D., Bourdev, L., Fergus, R., Torresani, L., Paluri, M.: Learning spatiotemporal features with 3D convolutional networks. In: 2015 IEEE International Conference on Computer Vision, ICCV, pp. 4489–4497. IEEE (2015)Google Scholar
  17. 17.
    Turner, R.H., Killian, L.M., et al.: Collective Behavior. Prentice-Hall, Englewood Cliffs (1957)Google Scholar
  18. 18.
    Urizar, O.J., Barakova, E.I., Marcenaro, L., Regazzoni, C.S., Rauterberg, M.: Emotion estimation in crowds: a survey (2017)Google Scholar
  19. 19.
    Wang, H., Kläser, A., Schmid, C., Liu, C.L.: Dense trajectories and motion boundary descriptors for action recognition. Int. J. Comput. Vis. 103(1), 60–79 (2013)MathSciNetCrossRefGoogle Scholar
  20. 20.
    Wang, J., Xu, Z.: Spatio-temporal texture modelling for real-time crowd anomaly detection. Comput. Vis. Image Underst. 144, 177–187 (2016)CrossRefGoogle Scholar
  21. 21.
    Xu, M., et al.: Crowd behavior simulation with emotional contagion in unexpected multi-hazard situations. arXiv preprint arXiv:1801.10000 (2018)
  22. 22.
    Yogameena, B., Nagananthini, C.: Computer vision based crowd disaster avoidance system: a survey. Int. J. Disaster Risk Reduct. 22, 95–129 (2017)CrossRefGoogle Scholar
  23. 23.
    Zhang, L., Feng, Y., Han, J., Zhen, X.: Realistic human action recognition: when deep learning meets VLAD. In: 2016 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP, pp. 1352–1356. IEEE (2016)Google Scholar
  24. 24.
    Zitouni, M.S., Bhaskar, H., Dias, J., Al-Mualla, M.E.: Advances and trends in visual crowd analysis: a systematic survey and evaluation of crowd modelling techniques. Neurocomputing 186, 139–159 (2016)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.Indian Institute of Information Technology and Management-Kerala (IIITM-K)ThiruvananthapuramIndia
  2. 2.Cochin University of Science and TechnologyKochiIndia

Personalised recommendations