Multimedia Tools and Applications

, Volume 77, Issue 23, pp 30633–30650 | Cite as

Affective image classification via semi-supervised learning from web images

  • Na Li
  • Yong XiaEmail author


Affective image classification has drawn increasing research attentions in the affective computing and multimedia communities. Despite many solutions proposed in the literature, it remains a major challenge to bridge the semantic gap between visual features of images and their affective characteristics, partly due to the lack of adequate training samples, which can be largely ascribed to the all-consuming nature of affective image annotation. In this paper, we propose a novel affective image classification algorithm based on semi-supervised learning from web images (SSL-WI). This algorithm consists of four major steps, including color and texture feature extraction, baseline classifier construction, feature selection, and jointly using training images and retrieved web images to re-train the classifier. We have applied this algorithm, the baseline classifier that is not trained by web images, and two state-of-the-art algorithms to differentiating color images in a three-dimensional discrete emotional space. Our results suggest that, with the scheme of semi-supervised learning from web images, the proposed algorithm is able to produce more accurate affective image classification than other three approaches.


Affective image classification Label propagate Support vector machine Adaboost Content-based image retrieval 



This work was supported in part by the National Natural Science Foundation of China under Grants 61771397 and 61471297, and in part by the Seed Foundation of Innovation and Creation for Graduate Students in Northwestern Polytechnical University under Grants Z2016150.


  1. 1.
    Baidu CBIR system-Baidu Institute of Deep Learning.
  2. 2.
    Bleschke M, Madonski R, Rudnicki R (2009) Image retrieval system based on combined MPEG-7 texture and colour descriptors. In: 2009 16th international conference on mixed design of integrated circuits & systems, pp 635–639Google Scholar
  3. 3.
    Bosse T, Zwanenburg E (2014) Do prospect-based emotions enhancebelievability of game characters? A casestudy in the context of a dice game. IEEE Trans Affect Comput 5(1):17–31CrossRefGoogle Scholar
  4. 4.
    Bovik AC, Clark M, Geisler WS (1990) Multichannel texture analysis using localized spatial filters. IEEE Trans Pattern Anal Mach Intell 12(1):55–73CrossRefGoogle Scholar
  5. 5.
    Chen H, Gao Z, Lu G, Li S (2008) A novel support vector machine fuzzy network for image classification using MPEG-7 visual descriptors. In: 2008 international conference on multimedia and information technology (MMIT), pp 365–368Google Scholar
  6. 6.
    Chen YY, Chen T, Liu T, Liao HYM, Chang SF (2015) Assistive image comment robot—a novel mid-level concept-based representation. IEEE Trans Affect Comput 6(3):298–311CrossRefGoogle Scholar
  7. 7.
    Cieplinski L, Kim M, Ohm JR, Pickering M, Yamada A (2001) Text of ISO/IEC 15938-3/FCD information technology multimedia content description interface part 3 visual: ISO/IEC JTC1/SC29/ WG11/N4062Google Scholar
  8. 8.
    Colombo C, Del Bimbo A, Pala P (1999) Semantics in visual information retrieval. IEEE MultiMed 6(3):38–53CrossRefGoogle Scholar
  9. 9.
    Cozman FG, Cohen I, Cirelo MC (2003) Semi-supervised learning of mixture models. In: Proceedings of the 20th international conference on machine learning, pp 99–106Google Scholar
  10. 10.
    Dai D, Van Gool L (2013) Ensemble projection for semi-supervised image classification. In: 2013 IEEE international conference on computer vision (ICCV), pp 2072–2079Google Scholar
  11. 11.
    Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR), vol 1, pp 886–893Google Scholar
  12. 12.
    Dorairaj R, Namuduri KR (2004) Compact combination of MPEG-7 color and texture descriptors for image retrieval. In: 2004 conference record of the thirty-eighth Asilomar conference on signals, systems and computers, vol 1, pp 387–391Google Scholar
  13. 13.
    Duan R, Jiang W, Man H (2006) Semi-supervised image classification in likelihood space. In: 2006 IEEE international conference on image processing (ICIP), pp 957–960Google Scholar
  14. 14.
    Geusebroek JM (2006) Compact object descriptors from local colour invariant histograms. Brit Mach Vis Conf 3:1029–1038Google Scholar
  15. 15.
    Houle ME, Oria V, Satoh SI, Sun J (2013) Annotation propagation in image databases using similarity graphs. ACM Trans Multimed Comput Commun Appl 10(1):1–21CrossRefGoogle Scholar
  16. 16.
    Huang K (2008) Colorful natural scenes retrieval based on affective features hierarchical model. In: 2008 international symposium on information science and engineering (ISISE), vol 2, pp 183–187Google Scholar
  17. 17.
    Itten J (1992) The art of color (Chinese version). Shanghai People’s Art Press, ShanghaiGoogle Scholar
  18. 18.
    Itten J (1992) Design and form-the basic course at the Bauhous (Chinese version). Shanghai People’s Art Press, ShanghaiGoogle Scholar
  19. 19.
    Joonwhoan L, EunJong P (2011) Fuzzy similarity-based emotional classification of color images. IEEE Trans Multimed 13(5):1031–1039CrossRefGoogle Scholar
  20. 20.
    Joshi D, Datta R, Fedorovskaya E, Luong QT (2011) Aesthetics and emotions in images. IEEE Signal Process Mag 28(5):94–115CrossRefGoogle Scholar
  21. 21.
    Kawamto N, Soen T (1993) Objective evaluation of color design II. Color Res Appl 18:260–266CrossRefGoogle Scholar
  22. 22.
    Le THN, Luu K, Savvides M (2013) SparCLeS: dynamic sparse classifiers with level sets for robust beard/moustache detection and segmentation. IEEE Trans Image Process 22Google Scholar
  23. 23.
    Lee PM, Hsiao TC (2014) Applying LCS to affective image classification in spatial-frequency domain. In: 2014 IEEE congress on evolutionary computation (CEC), pp 1690–1697Google Scholar
  24. 24.
    Li HF, Jin Q (2008) Research of image affective semantic rules based on neural network. In: 2008 international seminar on future biomedical information engineering (FBIE), pp 148–151Google Scholar
  25. 25.
    Li YF, Zhou ZH (2015) Towards making unlabeled data never hurt. IEEE Trans Pattern Anal Mach Intell 37(1):175–188CrossRefGoogle Scholar
  26. 26.
    Li HF, Li J, Song JC (2007) Extracting emotional semantics from color image using analytical hierarchy process. In: 2007 third international conference on intelligent information hiding and multimedia signal processing (IIHMSP), vol 1, pp 612–618Google Scholar
  27. 27.
    Li HF, Wen X, Jin H (2008) The clustering algorithm research of image emotional characteristics based on ant colony. In: 2008 international symposium on intelligent information technology application workshops (IITAW), pp 455–458Google Scholar
  28. 28.
    Li N, Xia Y, Xia Y (2015) Semi-supervised emotional classification of color images by learning from cloud. In: 2015 international conference on affective computing and intelligent interaction (ACII), pp 84–90Google Scholar
  29. 29.
    Liu Z, Yu X (2012) Identification of image emotional semantic based on feature fusion. In: 2012 international conference on computer science & service system (CSSS), pp 1802–1806Google Scholar
  30. 30.
    Liu W, Jiang Y-G, Luo J, Chang S-F (2011) Noise resistant graph ranking for improved web image search. In: 2011 IEEE conference on computer vision and pattern recognition (CVPR), pp 849–856Google Scholar
  31. 31.
    Machajdik J, Hanbury A (2010) Affective Imag classification using features inspired by psychology and art theory. In: Proceedings of the international conference on multimedia. ACM, pp 83–92Google Scholar
  32. 32.
    Merdassi H, Barhoumi W, Zagrouba E (2014) Semi-supervised image classification in large datasets by using random forest and fuzzy quantification of the salient object. In: 2014 international workshop on computational intelligence for multimedia understanding (IWCIM), pp 1–5Google Scholar
  33. 33.
    Ou LC, Luo MR, Woodcock A, Wright A (2010) A study of colour emotion and colour preference. Part I: colour emotions for single colours. Color Res Appl 29 (3):232–240CrossRefGoogle Scholar
  34. 34.
    Park TS, Zhang BT (2015) Consensus analysis and modeling of visual aesthetic perception. IEEE Trans Affect Comput 6(3):272–285CrossRefGoogle Scholar
  35. 35.
    Smith JR, Chang SF (1996) A fully automated content-based image query system. ACM MultimedGoogle Scholar
  36. 36.
    Soen T, Shimada T, Akita M (1987) Objective evaluation of color design. Color Res Appl 12:187–195CrossRefGoogle Scholar
  37. 37.
    Solli M, Lenz R (2011) Color emotions for multi-colored images. Color Res Appl 36(3):210–221CrossRefGoogle Scholar
  38. 38.
    Song X, Peng X, Xu J, Wu F (2014) Cloud-based distributed image coding. In: 2014 IEEE international conference on image processing (ICIP), pp 4802–4806Google Scholar
  39. 39.
    Sung-Bae C (2004) Emotional image and musical information retrieval with interactive genetic algorithm. Proc IEEE 92(4):702–711CrossRefGoogle Scholar
  40. 40.
    Tang J, Hong R, Yan S, Chua T-S, Qi G-J, Jain R (2011) Image annotation by k nn-sparse graph-based label propagation over noisily tagged web images. ACM Trans Intell Syst Technol (TIST) 2(2):14Google Scholar
  41. 41.
    Tang JH, Zha ZJ, Tao D, Chua TS (2012) Semantic-gap-oriented active learning for multilabel image annotation. IEEE Trans Image Process 21(4):2354–2360MathSciNetCrossRefGoogle Scholar
  42. 42.
    Tian GJ, Fu H, Feng DD (2008) Automatic medical image categorization and annotation using LBP and MPEG-7 edge histograms. 2008 international conference on information technology and applications in biomedicine (ITAB), pp 51–53Google Scholar
  43. 43.
    Tkalcic M, Odic A, Kosir A, Tasic J (2013) Affective labeling in a content-based recommender system for images. IEEE Trans Multimed 15(2):391–400CrossRefGoogle Scholar
  44. 44.
    Wang WN, He QH (2008) A survey on emotional semantic image retrieval. In: 2008 15th IEEE international conference on image processing (ICIP), pp 117–120Google Scholar
  45. 45.
    Wang J, Liu B (2010) A study on emotion classification of image based on bp neural network. In: 2010 international conference of information science and management engineering (ISME), vol 1, pp 100–104Google Scholar
  46. 46.
    Wang S, Wang X (2005) Emotion semantics image retrieval: an brief overview. In: 2005 international conference on affective computing and intelligent interaction (ACII), pp 490–497CrossRefGoogle Scholar
  47. 47.
    Wang WN, Yu YL (2005) Image emotional semantic query based on color semantic description. In: Proceedings of 2005 international conference on machine learning and cybernetics, vol 7, pp 4571–4576Google Scholar
  48. 48.
    Wang WN, Yu YL, Zhang JC (2004) Image emotional classification: static vs. dynamic. In: 2004 IEEE international conference on systems, man and cybernetics, vol 7, pp 6407–6411Google Scholar
  49. 49.
    Wang WN, Yu YL, Jiang SM (2006) Image retrieval by emotional semantics: a study of emotional space and feature extraction. In: 2006 IEEE international conference on systems, man and cybernetics (SMC), vol 4, pp 3534–3539Google Scholar
  50. 50.
    Wang X, Jia J, Yin J (2013) Interpretable aesthetic features for affective image classification. In: 2013 20th IEEE international conference on image processing (ICIP), pp 3230–3234Google Scholar
  51. 51.
    Wang Q, Meng Z, Li X (2017) Locality adaptive discriminant analysis for spectral–spatial classification of hyperspectral images. IEEE Geosci Remote Sens Lett (GRSL) 14(11):2077–2081CrossRefGoogle Scholar
  52. 52.
    Wang Q, Wan J, Yuan Y (2017) Deep metric learning for crowdedness regression. IEEE Trans Circuits Syst Video Technol (T-CSVT) 99:1–1Google Scholar
  53. 53.
    Wang Q, Wan J, Yuan Y (2018) Locality constraint distance metric learning for traffic congestion detection. Pattern Recogn 75:272–281CrossRefGoogle Scholar
  54. 54.
    Xie WX, Lu ZW, Peng YX, Xiao JG (2013) Multimodal semi-supervised image classification by combining tag refinement, graph-based learning and support vector regression. In: 2013 20th IEEE international conference on image processing (ICIP), pp 4307–4311Google Scholar
  55. 55.
    Yamada A, Pickering M, Jeannin S, Cieplinski L, Ohm JR, Kim M (2001) MPEG-7 visual part of experimentation model version 10.0: ISO/IEC JTC1/SC29/WG11/N4063Google Scholar
  56. 56.
    Yannakakis GN, Isbister K, Paiva A, Karpouzis K (2014) Guest editorial: emotion in games. IEEE Trans Affect Comput 5(1):1–2CrossRefGoogle Scholar
  57. 57.
    Yanulevskaya V, van Gemert JC, Roth K (2008) Emotional valence categorization using holistic image features. In: 2008 15th IEEE international conference on image processing (ICIP), pp 101–104Google Scholar
  58. 58.
    Zhang X, Ren F (2008) Improving Svm learning accuracy with Adaboost. In: 2008 fourth international conference on natural computation (ICNC), vol 3, pp 221–225Google Scholar
  59. 59.
    Zhao Y, Wang B, Ye S, Zheng Y, Shi L (2009) Image retrieval based on improved MCD and EHD of MPEG-7. In: 2009 international conference on test and measurement (ICTM), vol 1, pp 127–132Google Scholar
  60. 60.
    Zhao J, Lu H, Li Y, Chen J (2012) A kind of fuzzy decision tree based on the image emotion classification. In: 2012 international conference on computing, measurement, control and sensor network (CMCSN), pp 167–170Google Scholar
  61. 61.
    Zhao S, Gao Y, Jiang X, Yao H, Chua T-S, Sun X (2014) Exploring principles-of-art features for image emotion recognition. In: Proceedings of the 22nd ACM international conference on multimedia, pp 47–56Google Scholar
  62. 62.
    Zhao M, Zhan C, Wu Z, Tang P (2015) Semi-supervised image classification based on local and global regression. IEEE Signal Process Lett 22(10):1666–1670CrossRefGoogle Scholar
  63. 63.
    Zhao S, Han Y, Zou Q, Hu Q (2016) Hierarchical support vector machine based structural classification with fused hierarchies. Neurocomputing 214(Supplement C):86–92CrossRefGoogle Scholar
  64. 64.
    Zhu S, Sun X, Jin D (2016) Multi-view semi-supervised learning for image classification. Neurocomputing 208(Supplement C):136–142Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Shaanxi Key Lab of Speech & Image Information Processing (SAIIP), School of Computer Science and EngineeringNorthwestern Polytechnical UniversityXi’anChina
  2. 2.Centre for Multidisciplinary Convergence Computing (CMCC), School of Computer Science and EngineeringNorthwestern Polytechnical UniversityXi’anChina

Personalised recommendations