Visual Sentiment Analysis by Combining Global and Local Information

  • Lifang Wu
  • Mingchao Qi
  • Meng JianEmail author
  • Heng Zhang


With the development of visual social networks, the sentiment analysis of images has quickly emerged for opinion mining. Based on the observation that the sentiments conveyed by some images are related to salient objects in them, we propose a scheme for visual sentiment analysis that combines global and local information. First, the sentiment is predicted from the entire images. Second, it is judged whether there are salient objects in an image or not. If there are, sub-images are cropped from the entire image based on the detection window of the salient objects. Moreover, a CNN model is trained for the set of sub-images. Predictions of sentiments from entire images and sub-images are then fused together to obtain the final results. If no salient object is detected in the images, the sentiment predicted directly from entire images is used as the final result. The compared experimental results show that the proposed approach is superior to state-of-the-art algorithms. It also demonstrates that reasonably utilizing the local information could improve the performance for visual sentiment analysis.


Visual sentiment analysis Salient objects Local information Global information 



  1. 1.
    Zhao S, Yao H, Yue G, Ding G, Chua TS (2018) Predicting personalized image emotion perceptions in social networks. IEEE Trans Affect Comput 9:526–540. CrossRefGoogle Scholar
  2. 2.
    Yang J, Sun M, Sun X (2017) Learning visual sentiment distributions via augmented conditional probability neural network. In: AAAI Conference on Artificial Intelligence, North America, February 2017. Accessed 03 Apr 2019
  3. 3.
    Kosti R, Alvarez JM, Recasens A, Lapedriza A (2017) Emotion recognition in context. In: IEEE conference on computer vision and pattern recognition.
  4. 4.
    Zhao S, Yue G, Ding G, Chua TS (2017) Real-time multimedia social event detection in microblog. IEEE Trans Cybern. Google Scholar
  5. 5.
    Zhao S, Ding G, Huang Q, Chua TS, Bjorn WS, Kurt K (2018) Affective image content analysis: a comprehensive survey. In: IJCAIGoogle Scholar
  6. 6.
    Joshi D, Datta R, Fedorovskaya E, Luong QT (2011) Aesthetics and emotions in images. IEEE Signal Process Mag 28:94–115. CrossRefGoogle Scholar
  7. 7.
    Zhao S, Gao Y, Jiang X, Yao H, Chua TS, Sun X (2014) Exploring principles-of-art features for image emotion recognition. In: ACM int. conf. multimedia, pp 47–56.
  8. 8.
    Zhang H, Augilius E, Honkela T, Laaksonen J, Gamper H, Alene H (2011) Analyzing emotional semantics of abstract art using low-level image features. In: International conference on advances in intelligent data analysis X, LNCS, vol 7014, pp 413–423.
  9. 9.
    Li B, Feng S, Xiong W, Hu W (2012) Scaring or pleasing: exploit emotional impact of an image. In: ACM international conference on multimedia, pp 1365–1366.
  10. 10.
    Liu N, Dellandra E, Tellez B, Chen L (2011) Associating textual features with visual ones to improve affective image classification. Springer, Berlin, pp 229–238. Google Scholar
  11. 11.
    Lu X, Suryanarayan P, Li J, Newman MG, Wang JZ (2012) On shape and the computability of emotions. In: ACM international conference on multimedia, pp 229–238.
  12. 12.
    Yuan J, Mcdonough S, You Q, Luo J (2013) Sentribute: image sentiment analysis from a mid-level perspective. In: Proceedings of the second international workshop on issues of sentiment discovery and opinion mining.
  13. 13.
    Rao T, Xu M, Liu H, Wang J, Burnett I (2016) Multi-scale blocks based image emotion classification using multiple instance learning. IEEE Int Conf Image Process. Google Scholar
  14. 14.
    Borth D, Ji R, Chen T, Breuel T, Chang SF (2013) Large-scale visual sentiment ontology and detectors using adjective noun pairs. In: 21st ACM international conference on Multimedia, pp 223–232.
  15. 15.
    Zhao S, Yao H, Yue G, Ji R, Ding G (2017) Continuous probability distribution prediction of image emotions via multi-task shared sparse regression. IEEE Trans Multimed 19:632–645. CrossRefGoogle Scholar
  16. 16.
    Zhao S, Ding G, Yue G, Xin Z, Tang Y, Han J, Yao H, Huang Q (2018) Discrete probability distribution prediction of image emotions with shared sparse learning. IEEE Trans Affect Comput. Google Scholar
  17. 17.
    Xu C, Cetintas S, Lee KC, Li LJ (2014) Visual sentiment prediction with deep convolutional neural networks. Eprint ArXiv arXiv:1411.5731
  18. 18.
    You Q, Luo J, Jin H, Yang J (2015) Robust image sentiment analysis using progressively trained and domain transferred deep networks. In: Twenty-ninth AAAI conference on artificial intelligence, pp 381–388. arXiv:1509.06041
  19. 19.
    Wu L, Liu S, Jian M, Luo J, Zhang X, Qi M (2018) Reducing noisy labels in weakly labeled data for visual sentiment analysis. In: IEEE international conference on image processing, pp 1322–1326.
  20. 20.
    Zheng H, Chen T, Luo J (2016) When saliency meets sentiment: understanding how image content invokes emotion and sentiment, pp 630–634. arXiv:1611.04636
  21. 21.
    Fan S, Jiang M, Shen Z et al (2017) The role of visual attention in sentiment prediction. In: Proceedings of the 2017 ACM on multimedia conference. ACM, pp 217–225.
  22. 22.
    Fan S, Shen Z, Jiang M, Koenig B, Xu J, Kankanhalli M, Zhao Q (2018) Emotional attention: a study of image sentiment and visual attention. In: The IEEE conference on computer vision and pattern recognition (CVPR)Google Scholar
  23. 23.
    You Q, Jin H, Luo J (2017) Visual sentiment analysis by attending on local image regions. In: AAAI conf. artif. intell, pp 231–237Google Scholar
  24. 24.
    Yang J, She D, Sun M, Cheng MM, Rosin P, Wang L (2018) Visual sentiment prediction based on automatic discovery of affective regions. IEEE Trans Multimed 20:2513–2525. CrossRefGoogle Scholar
  25. 25.
    Zhang J, Sclaroff S, Lin Z, Shen X, Price B, Mech R (2016) Unconstrained salient object detection via proposal subset optimization. In: Computer vision and pattern recognition, pp 5733–5742.
  26. 26.
    You Q, Luo J, Jin H, Yang J (2016) Building a large scale dataset for image emotion recognition: the fine print and the benchmark. In: AAAI conf. artif. intel, pp 308–314. arXiv:1605.02677
  27. 27.
    Simonyan K, Zisserman (2014) A very deep convolutional networks for large-scale image recognition. Comput Sci. arXiv:1409.1556v6
  28. 28.
    Peng KC, Sadovnik A, Gallagher A, Chen T (2016) Where do emotions come from? Predicting the emotion stimuli map. In: IEEE international conference on image processing.
  29. 29.
    Chen T, Borth D, Darrell T, Chang SF (2014) Deep SentiBank: visual sentiment concept classification with deep convolutional neural networks. arXiv:1410.8586

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Faculty of Information TechnologyBeijing University of TechnologyBeijingChina

Personalised recommendations