A Novel Affective Visualization System for Videos Based on Acoustic and Visual Features

  • Jianwei NiuEmail author
  • Yiming Su
  • Shasha Mo
  • Zeyu Zhu
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10133)


With the fast development of social media in recent years, affective video content analysis has become a hot research topic and the relevant techniques are adopted by quite a few popular applications. In this paper, we firstly propose a novel set of audiovisual movie features to improve the accuracy of affective video content analysis, including seven audio features, eight visual features and two movie grammar features. Then, we propose an iterative method with low time complexity to select a set of more significant features for analyzing a specific emotion. And then, we adopt the BP (Back Propagation) network and circumplex model to map the low-level audiovisual features onto high-level emotions. To validate our approach, a novel video player with affective visualization is designed and implemented, which makes emotion visible and accessible to audience. Finally, we built a video dataset including 2000 video clips with manual affective annotations, and conducted extensive experiments to evaluate our proposed features, algorithms and models. The experimental results reveals that our approach outperforms state-of-the-art methods.


Affective analysis Novel features Feature selection Emotion visualization 



This work was supported by the National Natural Science Foundation of China (Grant Nos. 61572060, 61190125, 61472024) and CERNET Innovation Project 2015 (Grant No. NGII20151004).


  1. 1.
    Arifin, S., Cheung, P.: Affective level video segmentation by utilizing the pleature-arousal-dominance information. IEEE Trans. Multimed. 10(7), 1325–1341 (2008)CrossRefGoogle Scholar
  2. 2.
    Niu, J., Zhao, X., Zhu, L., Li, H.: Affivir: an affect-based Internet video recommendation system. Neurocomputing 120, 422–433 (2013)CrossRefGoogle Scholar
  3. 3.
    Arapakis, I., Moshfeghi, Y., Joho, H., Ren, R., Hannah, D., Jose, J.M.: Enriching user profiling with affective features for the improvement of a multimodal recommender system. In: Proceedings of the ACM International Conference on Image and Video Retrieval, p. 29. ACM (2009)Google Scholar
  4. 4.
    Lin, K.S., Lee, A., Yang, Y.H., Lee, C.T., Chen, H.: Automatic highlight extraction for drama video using music emotion and human face features. Neurocomputing 119, 111–117 (2013)CrossRefGoogle Scholar
  5. 5.
    Zhang, S., Huang, Q., Jiang, S., Gao, W., Tian, Q.: Affective visualization and retrieval for music video. IEEE Trans. Multimed. 12(6), 510–522 (2010)CrossRefGoogle Scholar
  6. 6.
    Hanjalic, A., Xu, L.Q.: Affective video content representation and modeling. IEEE Trans. Multimed. 7(1), 143–154 (2005)CrossRefGoogle Scholar
  7. 7.
    Chan, C.H., Jones, G.J.F.: An affect-based video retrieval system with open vocabulary querying. In: Detyniecki, M., Knees, P., Nürnberger, A., Schedl, M., Stober, S. (eds.) AMR 2010. LNCS, vol. 6817, pp. 103–117. Springer, Heidelberg (2012). doi: 10.1007/978-3-642-27169-4_8 CrossRefGoogle Scholar
  8. 8.
    Zhang, S., Tian, Q., Jiang, S., Gao, W.: Affective MTV analysis based on arousal and valence features. In: 2008 IEEE International Conference on Multimedia and Expo, pp. 1369–1372. IEEE (2008)Google Scholar
  9. 9.
    Canini, L., Benini, S., Leonardi, R.: Affective recommendation of movies based on selected connotative features. IEEE Trans. Circ. Syst. Video Technol. 23(4), 636–647 (2013)CrossRefGoogle Scholar
  10. 10.
    Xu, M., Wang, J., He, X., et al.: A three-level framework for affective content analysis and its case studies. Multimed. Tools Appl. 70(4), 757–779 (2014)CrossRefGoogle Scholar
  11. 11.
    Cui, Y., Jin, J.S., Zhang, S., Tian, Q.: Music video affective understanding using feature importance analysis. In: Proceedings of the ACM International Conference on Image and Video Retrieval, pp. 213–219. ACM (2010)Google Scholar
  12. 12.
    Yazdani, A., Kappeler, K., Ebrahimi, T.: Affective content analysis of music video clips. In: Proceedings of the 1st International ACM Workshop on Music Information Retrieval with User-Centered and Multimodal Strategies, pp. 7–12. ACM (2011)Google Scholar
  13. 13.
    Acar, E., Hopfgartner, F., Albayrak, S.: Understanding affective content of music videos through learned representations. In: Gurrin, C., Hopfgartner, F., Hurst, W., Johansen, H., Lee, H., O’Connor, N. (eds.) MMM 2014. LNCS, vol. 8325, pp. 303–314. Springer, Heidelberg (2014). doi: 10.1007/978-3-319-04114-8_26 CrossRefGoogle Scholar
  14. 14.
    Canini, L., Benini, S., Leonardi, R.: Affective analysis on patterns of shot types in movies. In: International Symposium on Image and Signal Processing and Analysis, pp. 253–258 (2011)Google Scholar
  15. 15.
    Xin, J.H., Cheng, K.M., Chong, T.F.: Quantifying colour emotion-what has been achieved. Res. J. Text. Apparel 2(1), 46–54 (1998)CrossRefGoogle Scholar
  16. 16.
    Russell, J.A.: A circumplex model of affect. J. Pers. Soc. Psychol. 39(6), 1161–1178 (1980)CrossRefGoogle Scholar
  17. 17.
    Lu, L., Liu, D., Zhang, H.: Automatic mood detection and tracking of music audio signals. IEEE Trans. Audio Speech Lang. Process. 14(1), 5–18 (2006)CrossRefGoogle Scholar
  18. 18.
    Valdez, P., Mehrabian, A.: Effects of color on emotions. J. Exp. Psychol. Gen. 123(4), 394–409 (1994)CrossRefGoogle Scholar
  19. 19.
    Wang, H.L., Cheong, L.-F.: Affective understanding in film. IEEE Trans. Circ. Syst. Video Technol. 16(6), 689–704 (2006)CrossRefGoogle Scholar
  20. 20.
    Baveye, Y., Dellandrea, E., Chamaret, C., Chen, L.: LIRIS-ACCEDE: a video database for affective content analysis. IEEE Trans. Affect. Comput. 6(1), 43–55 (2015)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.State Key Laboratory of Virtual Reality Technology and Systems, School of Computer Science and EngineeringBeihang UniversityBeijingChina
  2. 2.School of Electronics and InformationXi’an Jiaotong UniversityXi’anChina

Personalised recommendations