Advertisement

Multimedia Tools and Applications

, Volume 78, Issue 6, pp 6939–6967 | Cite as

A survey on sentiment analysis and opinion mining for social multimedia

  • Zuhe LiEmail author
  • Yangyu Fan
  • Bin Jiang
  • Tao Lei
  • Weihua Liu
Article
  • 519 Downloads

Abstract

Social media sentiment analysis (also known as opinion mining) which aims to extract people’s opinions, attitudes and emotions from social networks has become a research hotspot. Conventional sentiment analysis concentrates primarily on the textual content. However, multimedia sentiment analysis has begun to receive attention since visual content such as images and videos is becoming a new medium for self-expression in social networks. In order to provide a reference for the researchers in this active area, we give an overview of this topic and describe the algorithms of sentiment analysis and opinion mining for social multimedia. Having conducted a brief review on textual sentiment analysis for social media, we present a comprehensive survey of visual sentiment analysis on the basis of a thorough investigation of the existing literature. We further give a summary of existing studies on multimodal sentiment analysis which combines multiple media channels. We finally summarize the existing benchmark datasets in this area, and discuss the future research trends and potential directions for multimedia sentiment analysis. This survey covers 100 articles during 2008–2018 and categorizes existing studies according to the approaches they adopt.

Keywords

Sentiment analysis Opinion mining Social media Multimedia sentiment 

Notes

Acknowledgements

This work was supported by the National Natural Science Foundation of China under Grant 61702462, 61702464 and 61461025, the Scientific and Technological Project of Henan Province under Grant 182102210607, and the Science and Technology Innovation Engineering Program for Shaanxi Provincial Key Laboratories under Grant 2013SZS15-K02.

References

  1. 1.
    Abburi H, Shrivastava M, Gangashetty S V (2016) Improved multimodal sentiment detection using stressed regions of audio. In: Proceedings of IEEE Region 10 Conference (TENCON), 2834–2837Google Scholar
  2. 2.
    Ahsan U, De Choudhury M, Essa I (2017) Towards using visual attributes to infer image sentiment of social events. In: Proceedings of International Joint Conference on Neural Networks (IJCNN), 1372–1379Google Scholar
  3. 3.
    Amencherla M, Varshney L R (2017) Color-based visual sentiment for social communication. In: Proceedings of 15th Canadian Workshop on Information Theory (CWIT), Article number: 7994829Google Scholar
  4. 4.
    Amiriparian S, Cummins N, Ottl S, et al (2017) Sentiment analysis using image-based deep spectrum features. In: Proceedings of 7th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW), 26–29Google Scholar
  5. 5.
    Borth D, Ji R, Chen T, et al (2013) Large-scale visual sentiment ontology and detectors using adjective noun pairs. In: Proceedings of the ACM International Conference on Multimedia, 223–232Google Scholar
  6. 6.
    Borth D, Chen T, Ji R, et al (2013) SentiBank: large-scale ontology and classifiers for detecting sentiment and emotions in visual content. In: Proceedings of the ACM International Conference on Multimedia, 459–460Google Scholar
  7. 7.
    Cai G, Xia B (2015) Convolutional neural networks for multimedia sentiment analysis. In: Proceedings of Natural Language Processing and Chinese Computing, 159–167CrossRefGoogle Scholar
  8. 8.
    Cai Z, Cao D, Lin D, et al (2016) A spatial-temporal visual mid-level ontology for GIF sentiment analysis. In: Proceedings of IEEE Congress on Evolutionary Computation, 4860–4865Google Scholar
  9. 9.
    Cambria E (2016) Affective computing and sentiment analysis. IEEE Intell Syst 31(2):102–107CrossRefGoogle Scholar
  10. 10.
    Campos V, Salvador A, Giró-i-Nieto X, et al (2015) Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction. In: Proceedings of the 1st International Workshop on Affect & Sentiment in Multimedia, 57–62Google Scholar
  11. 11.
    Campos V, Jou B, Giró-i-Nieto X (2017) From pixels to sentiment: fine-tuning CNNs for visual sentiment prediction. Image Vis Comput 65:15–22CrossRefGoogle Scholar
  12. 12.
    Cao D, Ji R, Lin D et al (2016) Visual sentiment topic model based microblog image sentiment analysis. Multimed Tools Appl 75(15):8955–8968CrossRefGoogle Scholar
  13. 13.
    Cao D, Ji R, Lin D et al (2016) A cross-media public sentiment analysis system for microblog. Multimedia Systems 22(4):479–486CrossRefGoogle Scholar
  14. 14.
    Chen T, Yu F X, Chen J, et al (2014) Object-based visual sentiment concept analysis and application. In: Proceedings of the 2014 ACM International Conference on Multimedia, 367–376Google Scholar
  15. 15.
    Chen Y Y, Chen T, Hsu W H, et al (2014) Predicting viewer affective comments based on image content in social media. In: Proceedings of the ACM International Conference on Multimedia Retrieval, 233–240Google Scholar
  16. 16.
    Chen T, Borth D, Darrell T, et al (2014) DeepSentiBank: visual sentiment concept classification with deep convolutional neural networks. arXiv preprint arXiv: 1410.8586Google Scholar
  17. 17.
    Chen F, Gao Y, CAO D (2015) Multimodal hypergraph learning for microblog sentiment prediction. In: Proceedings of 2015 IEEE International Conference on Multimedia and Expo, 1–6Google Scholar
  18. 18.
    Chen S, Yang J, Feng J, et al (2017) Image sentiment analysis using supervised collective matrix factorization. In: Proceedings of 12th IEEE Conference on Industrial Electronics and Applications (ICIEA), 1033–1038Google Scholar
  19. 19.
    Chen M, Zhang L, Yu X et al (2017) Weighted co-training for cross-domain image sentiment classification. J Comput Sci Technol 32(4):714–725CrossRefGoogle Scholar
  20. 20.
    Chisholm D, Siddiquie B, Divakaran A, et al (2015) Audio-based affect detection in web videos. In: Proceedings of IEEE International Conference on Multimedia and Expo, 1–6Google Scholar
  21. 21.
    Chu E, Roy D (2017) Audio-visual sentiment analysis for learning emotional arcs in movies. In: Proceedings of 17th IEEE International Conference on Data Mining (ICDMW), 829–834Google Scholar
  22. 22.
    Cui A, Zhang M, Liu Y, et al (2011) Emotion tokens: Bridging the gap among multilingual twitter sentiment analysis. In Proceedings of the 7th Asia Conference on Information Retrieval Technology (AIRS’11), 238–249CrossRefGoogle Scholar
  23. 23.
    Cui J, Liu Y, Xu Y et al (2013) Tracking generic human motion via fusion of low- and high-dimensional approaches. IEEE Trans Syst, Man, Cybernetics: Syst 43(4):996–1002CrossRefGoogle Scholar
  24. 24.
    Da Silva NFF, Hruschka ER, Hruschka ER (2014) Tweet sentiment analysis with classifier ensembles. Decis Support Syst 66:170–179CrossRefGoogle Scholar
  25. 25.
    Dashtipour K, Poria S, Hussain A et al (2016) Multilingual sentiment analysis: state of the art and independent comparison of techniques. Cogn Comput 8(4):757–771CrossRefGoogle Scholar
  26. 26.
    Ding X, Liu B, Yu P S (2008) A holistic lexicon-based approach to opinion mining. In: Proceedings of the 2008 International Conference on Web Search and Data Mining (WSDM’08), 231–240Google Scholar
  27. 27.
    Dong L, Wei F, Tan C, et al (2014) Adaptive recursive neural network for target-dependent twitter sentiment classification. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, 49–54Google Scholar
  28. 28.
    Ezzat S, Gayar N, Ghanem M (2012) Sentiment analysis of call Centre audio conversations using text classification. Int J Comput Inform Syst Indust Manag Appl 4(2012):619–627Google Scholar
  29. 29.
    Fan Y, Li Z, Wang F et al (2017) Affective abstract image classification based on convolutional sparse autoencoders across different domains. J Electron Inf Technol 39(1):167–175Google Scholar
  30. 30.
    Ghiassi M, Skinner J, Zimbra D (2013) Twitter brand sentiment analysis: a hybrid system using n-gram analysis and dynamic artificial neural network. Expert Syst Appl 40(16):6266–6282CrossRefGoogle Scholar
  31. 31.
    Giachanou A, Crestani F (2016) Like it or not: a survey of twitter sentiment analysis methods. ACM Comput Surv 49(2):28CrossRefGoogle Scholar
  32. 32.
    Go A, Bhayani R, Huang L (2009) Twitter sentiment classification using distant supervision. CS224N Project Report, Stanford 1: 12Google Scholar
  33. 33.
    Hu X, Tang L, Tang J, et al (2013) Exploiting social relations for sentiment analysis in microblogging. In: Proceedings of the 6th ACM International Conference on Web Search and Data Mining (WSDM’13), 537–546Google Scholar
  34. 34.
    Ji R, Cao D, Lin D (2015) Cross-modality sentiment analysis for social multimedia. In: Proceedings of IEEE International Conference on Multimedia Big Data, 28–31Google Scholar
  35. 35.
    Ji R, Cao D, Zhou Y et al (2016) Survey of visual sentiment prediction for social media analysis. Front Comput Sci 10(4):602–611CrossRefGoogle Scholar
  36. 36.
    Jia J, Wu S, Wang X, et al (2012) Can we understand van Gogh's mood? Learning to infer affects from images in social networks. In: Proceedings of the 20th ACM International Conference on Multimedia, 857–860Google Scholar
  37. 37.
    Jiang Y, Xu B, Xue X (2014) Predicting emotions in user-generated videos. In: Proceedings of the 28th AAAI Conference on Artificial Intelligence and the 26th Innovative Applications of Artificial Intelligence Conference and the 5th Symposium on Educational Advances in Artificial Intelligence, 73–79Google Scholar
  38. 38.
    Jindal S, Singh S (2015) Image sentiment analysis using deep convolutional neural networks with domain specific fine tuning. In: Proceedings of 2015 International Conference on Information Processing, 447–451Google Scholar
  39. 39.
    Joshi D, Datta R, Fedorovskaya E et al (2011) Aesthetics and emotions in images. IEEE Signal Process Mag 28(5):94–115CrossRefGoogle Scholar
  40. 40.
    Jou B, Bhattacharya S, Chang SF (2014) Predicting viewer perceived emotions in animated GIFs. In: Proceedings of the ACM International Conference on Multimedia, 213–216Google Scholar
  41. 41.
    Jou B, Chen T, Pappas N, et al (2015) Visual affect around the world: A large-scale multilingual visual sentiment ontology. In: Proceedings of the 23rd ACM international conference on Multimedia, 159–168Google Scholar
  42. 42.
    Kaushik L, Sangwan A, Hansen J H L (2013) Sentiment extraction from natural audio streams. In: Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing, 8485–8489Google Scholar
  43. 43.
    Kaushik L, Sangwan A, Hansen J H L (2013) Automatic sentiment extraction from YouTube videos. In: Proceedings of 2013 IEEE Workshop on Automatic Speech Recognition and Understanding, 239–244Google Scholar
  44. 44.
    Kaushik L, Sangwan A, Hansen JHL (2015) Automatic audio sentiment extraction using keyword spotting. In: Proceedings of the 16th Annual Conference of the International Speech Communication Association, 2709–2713Google Scholar
  45. 45.
    Kaushik L, Sangwan A, Hansen JHL (2017) Automatic sentiment detection in naturalistic audio. IEEE-ACM Trans Audio, Speech, Language Proc 25(8):1668–1679CrossRefGoogle Scholar
  46. 46.
    Khan FH, Bashir S, Qamar U (2014) TOM: twitter opinion mining framework using hybrid classification scheme. Decis Support Syst 57:245–257CrossRefGoogle Scholar
  47. 47.
    Kiritchenko S, Zhu X, Mohammad SM (2014) Sentiment analysis of short informal text. J Artif Intell Res 50:723–762CrossRefGoogle Scholar
  48. 48.
    Kontopoulos E, Berberidis C, Dergiades T et al (2013) Ontology-based sentiment analysis of twitter posts. Expert Syst Appl 40(10):4065–4074CrossRefGoogle Scholar
  49. 49.
    Korenek P, Šimko M (2014) Sentiment analysis on microblog utilizing appraisal theory. World Wide Web 17(4):847–867CrossRefGoogle Scholar
  50. 50.
    Li Z, Fan Y (2015) Survey on visual sentiment analysis. Appl Res Comput 32(12):3521–3526MathSciNetGoogle Scholar
  51. 51.
    Li Z, Fan Y, Liu W (2015) The effect of whitening transformation on pooling operations in convolutional autoencoders. EURASIP J Adv Sign Proc 2015:37CrossRefGoogle Scholar
  52. 52.
    Li L, Cao D, Li S, et al (2015) Sentiment analysis of Chinese micro-blog based on multi-modal correlation model. In: Proceedings of the 2015 IEEE International Conference on Image Processing, 4798–4802Google Scholar
  53. 53.
    Li Z, Fan Y, Liu W et al (2017) Emotional textile image classification based on cross-domain convolutional sparse autoencoders with feature selection. J Electron Imaging 26(1):013022CrossRefGoogle Scholar
  54. 54.
    Li L, Li S, Cao D, et al (2017) SentiNet: Mining visual sentiment from scratch. In: Advances in Computational Intelligence Systems, 309–317Google Scholar
  55. 55.
    Li Z, Fan Y, Liu W et al (2018) Image sentiment prediction based on textual descriptions with adjective noun pairs. Multimed Tools Appl 77(1):1115–1132CrossRefGoogle Scholar
  56. 56.
    Lin J, Kolcz A (2012) Large-scale machine learning at twitter. In: Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data, 793–804Google Scholar
  57. 57.
    Lin D, Cao D, Lv Y et al (2017) GIF video sentiment detection using semantic sequence. Math Probl Eng 2017:6863174Google Scholar
  58. 58.
    Liu Y, Cui J, Zhao H, et al (2012) Fusion of low-and high-dimensional approaches by trackers sampling for generic human motion tracking. In: Proceedings of the 21st International Conference on Pattern Recognition, 898–901Google Scholar
  59. 59.
    Liu Y, Nie L, Han L, et al (2015) Action2Activity: Recognizing complex activities from sensor data. In: Proceedings of the 24th International Joint Conference on Artificial Intelligence 1617–1623Google Scholar
  60. 60.
    Liu L, Cheng L, Liu Y, et al (2016) Recognizing complex activities by a probabilistic interval-based model. In: Proceedings of the 30th AAAI Conference on Artificial Intelligence, AAAI 2016, 1266–1272Google Scholar
  61. 61.
    Liu Y, Nie L, Liu L et al (2016) From action to activity: sensor-based activity recognition. Neurocomputing 181:108–115CrossRefGoogle Scholar
  62. 62.
    Liu Y, Zhang L, Nie L, et al (2016) Fortune teller: Predicting your career path. In: Proceedings of the 30th AAAI Conference on Artificial Intelligence, AAAI 2016, 201–207Google Scholar
  63. 63.
    Liu H, Jou B, Chen T, et al (2016) Complura: Exploring and leveraging a large-scale multilingual visual sentiment ontology. In: Proceedings of the 2016 ACM International Conference on Multimedia Retrieval, 417–420Google Scholar
  64. 64.
    Lu Y, Wei Y, Liu L et al (2017) Towards unsupervised physical activity recognition using smartphone accelerometers. Multimed Tools Appl 76(8):10701–10719CrossRefGoogle Scholar
  65. 65.
    Maas A L, Daly R E, Pham P T, et al (2011) Learning word vectors for sentiment analysis. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, 142–150Google Scholar
  66. 66.
    Machajdik J, Hanbury A (2010) Affective image classification using features inspired by psychology and art theory. Proceedings of the 2010 ACM International Conference on Multimedia, 83–92Google Scholar
  67. 67.
    Morency L P, Mihalcea R, Doshi P (2011) Towards multimodal sentiment analysis: harvesting opinions from the web. In: Proceedings of the 2011 ACM International Conference on Multimodal Interaction, 169–176Google Scholar
  68. 68.
    Naaman M (2012) Social multimedia: highlighting opportunities for search and mining of multimedia data in social media applications. Multimed Tools Appl 56(1):9–34CrossRefGoogle Scholar
  69. 69.
    Narihira T, Borth D, Yu S X, et al (2015) Mapping images to sentiment adjective noun pairs with factorized neural nets. arXiv preprint arXiv:1511.06838Google Scholar
  70. 70.
    Niu T, Zhu S, Pang L, et al (2016) Sentiment analysis on multi-view social data. In: Proceedings of International Conference on Multimedia Modeling, 15–27Google Scholar
  71. 71.
    Poria S, Cambria E, Howard N et al (2016) Fusing audio, visual and textual clues for sentiment analysis from multimodal content. Neurocomputing 174:50–59CrossRefGoogle Scholar
  72. 72.
    Preotiuc-Pietro D, Hopkins D, Liu Y, et al (2017) Beyond binary labels: political ideology prediction of twitter users. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, 729–740Google Scholar
  73. 73.
    Ravi K, Ravi V (2015) A survey on opinion mining and sentiment analysis: tasks, approaches and applications. Knowl-Based Syst 89(C):14–46CrossRefGoogle Scholar
  74. 74.
    Rosas V, Mihalcea R, Morency LP (2013) Multimodal sentiment analysis of Spanish online videos. IEEE Intell Syst 28(3):38–45CrossRefGoogle Scholar
  75. 75.
    Sager S, Borth D, Elizalde B, et al (2016) AudioSentibank: Large-scale semantic ontology of acoustic concepts for audio content analysis. arXiv preprint arXiv:1607.03766Google Scholar
  76. 76.
    Saif H, He Y, Fernandez M et al (2016) Contextual semantics for sentiment analysis of twitter. Inf Process Manag 52(1):5–19CrossRefGoogle Scholar
  77. 77.
    Siddiquie B, Chisholm D, Divakaran A (2015) Exploiting multimodal affect and semantics to identify politically persuasive web videos. In: Proceedings of the 2015 ACM International Conference on Multimodal Interaction, 203–210Google Scholar
  78. 78.
    Siersdorfer S, Minack E, Deng F, et al (2010) Analyzing and predicting sentiment of images on the social web, In: Proceedings of the 2010 ACM International Conference on Multimedia, 715–718Google Scholar
  79. 79.
    Silva NFFD, Coletta LFS, Hruschka ER (2016) A survey and comparative study of tweet sentiment analysis via semi-supervised learning. ACM Comput Surv 49(1):15Google Scholar
  80. 80.
    Speriosu M, Sudan N, Upadhyay S, et al (2011) Twitter polarity classification with label propagation over lexical links and the follower graph. In: Proceedings of the First workshop on Unsupervised Learning in NLP, 53–63Google Scholar
  81. 81.
    Sun M, Yang J, Wang K, et al (2016) Discovering affective regions in deep convolutional neural networks for visual sentiment prediction. In: Proceedings of 2016 IEEE International Conference on Multimedia and Expo, 1–6Google Scholar
  82. 82.
    Taboada M, Brooke J, Tofiloski M et al (2011) Lexicon-based methods for sentiment analysis. Comput Linguistics 37(2):267–307CrossRefGoogle Scholar
  83. 83.
    Tang D, Wei F, Yang N, et al (2014) Learning sentiment-specific word embedding for twitter sentiment classification. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL’14), 1555–1565Google Scholar
  84. 84.
    Tang D, Qin B, Liu T (2015) Deep learning for sentiment analysis: successful approaches and future challenges. Wiley Interdiscip Rev Data Mining Knowledge Discovery 5(6):292–303CrossRefGoogle Scholar
  85. 85.
    Tang D, Qin B, Liu T (2015) Learning semantic representations of users and products for document level sentiment classification. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, 1014–1023Google Scholar
  86. 86.
    Tang D, Qin B, Liu T, et al (2015) User modeling with neural network for review rating prediction. In: Proceedings of the 24th International Conference on Artificial Intelligence (IJCAI’15), 1340–1346Google Scholar
  87. 87.
    Thelwall M, Buckley K, Paltoglou G et al (2010) Sentiment strength detection in short informal text. J Assoc Inform Sci Technol 61(12):2544–2558CrossRefGoogle Scholar
  88. 88.
    Vadicamo L, Carrara F, Cimino A, et al (2017) Cross-media learning for image sentiment analysis in the wild. In: Proceedings of 16th IEEE International Conference on Computer Vision (ICCV), 308–317Google Scholar
  89. 89.
    Wang W, He Q (2008) A survey on emotional semantic image retrieval. In: Proceedings of 15th IEEE International Conference on Image Processing, 117–120Google Scholar
  90. 90.
    Wang X, Wei F, Liu X, et al (2011) Topic sentiment analysis in twitter: a graph-based hashtag sentiment classification approach. In: Proceedings of the 20th ACM international conference on Information and knowledge management, 1031–1040Google Scholar
  91. 91.
    Wang Y, Wang S, Tang J, et al (2015) Unsupervised sentiment analysis for social media images. In: Proceedings of the 24th International Joint Conference on Artificial Intelligence, 2378–2379Google Scholar
  92. 92.
    Wang J, Fu J, Xu Y, et al (2016) Beyond object recognition: Visual sentiment analysis with deep coupled adjective and noun neural networks. In: Proceedings of the 25th International Joint Conference on Artificial Intelligence, 3484–3490Google Scholar
  93. 93.
    Yadav S K, Bhushan M, Gupta S (2015) Multimodal sentiment analysis: Sentiment analysis using audiovisual format. In: Proceedings of 2nd International Conference on Computing for Sustainable Global Development, 1415–1419Google Scholar
  94. 94.
    You Q (2016) Sentiment and emotion analysis for social multimedia: Methodologies and applications. In: Proceedings of the ACM International Conference on Multimedia, 1445–1449Google Scholar
  95. 95.
    You Q, Luo J, Jin H, et al (2015) Robust image sentiment analysis using progressively trained and domain transferred deep networks. In: Proceedings of the 29th AAAI Conference on Artificial Intelligence, 381–388Google Scholar
  96. 96.
    You Q, Luo J, Jin H, et al (2015) Joint visual-textual sentiment analysis with deep neural networks. In: Proceedings of ACM International Conference on Multimedia, 1071–1074Google Scholar
  97. 97.
    You Q, Luo J, Jin H, et al (2016) Cross-modality consistent regression for joint visual-textual sentiment analysis of social multimedia. In Proceedings of ACM International Conference on Web Search and Data Mining, 13–22Google Scholar
  98. 98.
    Yu Y, Lin H, Meng J et al (2016) Visual and textual sentiment analysis of a microblog using deep convolutional neural networks. Algorithms 9(2):41MathSciNetCrossRefGoogle Scholar
  99. 99.
    Yuan J, Mcdonough S, You Q, et al (2013) Sentribute: image sentiment analysis from a mid-level perspective. In: Proceedings of the Second International Workshop on Issues of Sentiment Discovery and Opinion Mining, Article number: 10Google Scholar
  100. 100.
    Zhang L, Ghosh R, Dekhil M, et al (2011) Combining lexicon-based and learning-based methods for twitter sentiment analysis. Hp Laboratories Technical ReportGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.School of Computer and Communication EngineeringZhengzhou University of Light IndustryZhengzhouChina
  2. 2.School of Electronics and InformationNorthwestern Polytechnical UniversityXi’anChina
  3. 3.College of Electronical and Information EngineeringShaanxi University of Science and TechnologyXi’anChina
  4. 4.Xi’an Institute of Optics and Precision MechanicsChinese Academy of SciencesXi’anChina

Personalised recommendations