Skip to main content
SpringerLink
Log in

Photo aesthetic quality estimation using visual complexity features

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

The amount of visual data available on the Web is growing explosively and it is becoming increasingly important to explore methods for automatically estimating the quality of this content in a manner that is consistent with the aesthetic perceptions of humans. The key to this challenging problem is to design an appropriate set of features to extract the aesthetic properties from content. Most previous studies designed a set of aesthetic features based on photographic criteria, which were unavoidably limited to specific examples and they lacked an interpretation based on the mechanism of human aesthetic perception. According to psychological theory, visual complexity is an important property of the stimuli, because it directly influences the viewer’s arousal level, which is believed to be closely related to aesthetic perception. In this study, we propose an alternative set of features for aesthetic estimation based on a visual complexity principle. We extracted the visual complexity properties from an input image in terms of their composition, shape, and distribution. In addition, we demonstrated that the proposed features are consistent with human perception on the complexity in our visual complexity dataset. Next, we employed these features for photo-aesthetic quality estimation using a large-scale dataset. Various experiments were conducted under different conditions and comparisons with state-of-the-art methods shows that the proposed visual complexity feature outperforms photography rule-based features and even better than deep features.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Notes

  1. http://www.lucamarchesotti.com/ava/

  2. Prediction results webpage link: https://www.hal.t.u-tokyo.ac.jp/~sun1101/

References

  1. Akalin A, Yildirim K, Wilson C, Kilicoglu O (2009) Architecture and engineering students’ evaluations of house faċades: preference, complexity and impressiveness. J Environ Psychol 29(1):124–132

    Article  Google Scholar 

  2. Arbelaez P, Maire M, Fowlkes C, Malik J (2011) Contour detection and hierarchical image segmentation. IEEE Trans Pattern Anal Mach Intell 33(5):898–916

    Article  Google Scholar 

  3. Babin BJ, Attaway JS (2000) Atmospheric affect as a tool for creating value and gaining share of customer. J Bus Res 49(2):91–99

    Article  Google Scholar 

  4. Berlyne D (1971) Aesthetics and psychobiology. Appleton-Century-Crofts, New York

    Google Scholar 

  5. Berlyne DE (1974) Studies in the new experimental aesthetics: Steps toward an objective psychology of aesthetic appreciation Hemisphere

  6. Bhattacharya S, Sukthankar R, Shah M (2010) A framework for photo-quality assessment and enhancement based on visual aesthetics Proceedings of the international conference on multimedia, pp 271–280

    Google Scholar 

  7. Birkhoff GD (1933) Aesthetic measure. Cambridge Mass

  8. Campbell A, Ciesielksi V, Qin A (2015) Feature discovery by deep learning for aesthetic analysis of evolved abstract images Evolutionary and biologically inspired music, sound, art and design. Springer, pp 27–38

  9. Chipman SF (1977) Complexity and structure in visual patterns. J Exp Psychol Gen 106(3):269

    Article  Google Scholar 

  10. Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection 2005 IEEE Computer society conference on computer vision and pattern recognition (CVPR’05), vol 1, pp 886–893

  11. Datta R, Joshi D, Li J, Wang JZ (2006) Studying aesthetics in photographic images using a computational approach Computer vision–ECCV 2006. Springer, pp 288–301

  12. Datta R, Li J, Wang JZ (2008) Algorithmic inferencing of aesthetics and emotion in natural images: an exposition Image processing, 2008. ICIP 2008. 15th IEEE international conference on, pp 105–108

    Chapter  Google Scholar 

  13. Dhar S, Ordonez V, Berg TL (2011) High level describable attributes for predicting aesthetics and interestingness Computer vision and pattern recognition (CVPR), 2011 IEEE conference on, pp 1657–1664

    Google Scholar 

  14. Donderi DC (2006) Visual complexity: a review. Psychol Bull 132(1):73

    Article  Google Scholar 

  15. Eysenck HJ (1942) The experimental study of the’good gestalt’—a new approach. Psychol Rev 49(4):344

    Article  Google Scholar 

  16. Friedenberg J, Bertamini M (2015) Aesthetic preference for polygon shape. Empir Stud Arts 33(2):144–160

    Article  Google Scholar 

  17. García M, Badre AN, Stasko JT (1994) Development and validation of icons varying in their abstractness. Interact Comput 6(2):191–211

    Article  Google Scholar 

  18. Guo X, Asano CM, Asano A, Kurita T, Li L (2012) Analysis of texture characteristics associated with visual complexity perception. Opt Rev 19(5):306–314

    Article  Google Scholar 

  19. He XC, Yung NH (2004) Curvature scale space corner detector with adaptive threshold and dynamic region of support Pattern recognition, 2004. ICPR 2004. Proceedings of the 17th international conference on, vol 2, pp 791–794

  20. Heaps C, Handel S (1999) Similarity and features of natural textures. J Exp Psychol Hum Percept Perform 25(2):299

    Article  Google Scholar 

  21. Huhmann BA (2003) Visual complexity in banner ads: The role of color, photography, and animation. Vis Commun Q 10(3):10–17

    Article  Google Scholar 

  22. Ichikawa S (1985) Quantitative and structural factors in the judgment of pattern complexity. Percept Psychophys 38(2):101–109

    Article  Google Scholar 

  23. Imamoglu Ç (2000) Complexity, liking and familiarity: Architecture and non-architecture turkish students’assessments of traditional and modern house facades. J Environ Psychol 20(1):5–16

    Article  Google Scholar 

  24. Katz BF (2002) What makes a polygon pleasing? Empir Stud Arts 20(1):1–19

    Article  Google Scholar 

  25. Ke Y, Tang X, Jing F (2006) The design of high-level features for photo quality assessment Computer vision and pattern recognition, 2006 IEEE computer society conference on, vol 1, pp 419–426

  26. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks Advances in neural information processing systems, pp 1097–1105

    Google Scholar 

  27. Leder H, Belke B, Oeberst A, Augustin D (2004) A model of aesthetic appreciation and aesthetic judgments. Br J Psychol 95(4):489–508

    Article  Google Scholar 

  28. Li C, Loui AC, Chen T (2010) Towards aesthetics: a photo quality assessment and photo selection system Proceedings of the international conference on multimedia, pp 827–830

    Google Scholar 

  29. Lihua G, Fudi L (2015). arXiv:1505.05225

  30. Lo KY, Liu KH, Chen CS (2012) Assessment of photo aesthetics with efficiency Pattern recognition (ICPR), 2012 21st international conference on, pp 2186–2189

    Google Scholar 

  31. Lu P, Peng X, Li R, Wang X (2015) Towards aesthetics of image: a bayesian framework for color harmony modeling. Signal Processing: Image Communication

  32. Lu X, Lin Z, Jin H, Yang J, Wang JZ (2014) Rapid: Rating pictorial aesthetics using deep learning Proceedings of the ACM international conference on multimedia, pp 457–466

    Google Scholar 

  33. Lu X, Lin Z, Shen X, Mech R, Wang JZ (2015) Deep multi-patch aggregation network for image style, aesthetics, and quality estimation Proceedings of the IEEE international conference on computer vision, pp 990–998

    Google Scholar 

  34. Luo Y, Tang X (2008) Photo and video quality evaluation: Focusing on the subject Computer vision–ECCV 2008. Springer, pp 386–399

  35. Luo W, Wang X, Tang X (2011) Content-based photo quality assessment Computer vision (ICCV), 2011 IEEE international conference on, pp 2206–2213

    Chapter  Google Scholar 

  36. Mai L, Jin H, Liu F (2016) Composition-preserving deep photo aesthetics assessment Proceedings of the IEEE conference on computer vision and pattern recognition, pp 497–506

    Google Scholar 

  37. Mallon B, Redies C, Hayn-Leichsenring GU (2014) Beauty in abstract paintings: perceptual contrast and statistical properties. Frontiers in human neuroscience 8

  38. Martín H JA, Santos M, de Lope J (2010) Orthogonal variant moments features in image analysis. Inf Sci 180(6):846–860

  39. Marchesotti L, Perronnin F, Larlus D, Csurka G (2011) Assessing the aesthetic quality of photographs using generic image descriptors Computer vision (ICCV), 2011 IEEE international conference on, pp 1784–1791

    Chapter  Google Scholar 

  40. Munsinger H, Kessen W (1964) Uncertainty, structure, and preference. Psychological Monogr: Gen Appl 78(9):1

    Article  Google Scholar 

  41. Murray N, Marchesotti L, Perronnin F (2012) Ava: a large-scale database for aesthetic visual analysis Computer vision and pattern recognition (CVPR), 2012 IEEE conference on, pp 2408–2415

    Chapter  Google Scholar 

  42. Nadal M, Besch MN, Kelley CS, Mains K, Marty G (2008) The influence of visual complexity on aesthetic preference: An explanation of diverging results. Proceedings of IAEA08 pp 137–141

  43. Nadal M, Munar E, Marty G, Cela-Conde CJ (2010) Visual complexity and beauty appreciation: Explaining the divergence of results. Empir Stud Arts 28 (2):173–191

    Article  Google Scholar 

  44. Nasar JL (2002) What design for a presidential library? Complexity, typicality, order, and historical significance. Empir Stud Arts 20(1):83–99

    Article  Google Scholar 

  45. Nishiyama M, Okabe T, Sato I, Sato Y (2011) Aesthetic quality classification of photographs based on color harmony Computer vision and pattern recognition (CVPR), 2011 IEEE conference on, pp 33–40

    Google Scholar 

  46. O’Donovan P, Agarwala A, Hertzmann A (2011) Color compatibility from large datasets ACM Transactions on graphics (TOG), vol 30-4, p 63. ACM

    Google Scholar 

  47. Oliva A, Mack ML, Shrestha M, Peeper A (2004) Identifying the perceptual dimensions of visual complexity of scenes Proc. of the 26th annual meeting of the cogn. Sci. soc, pp 101–106

    Google Scholar 

  48. Palumbo L, Bertamini M (2016) The curvature effect a comparison between preference tasks. Empir Stud Arts 34(1):35–52

    Article  Google Scholar 

  49. Redies C, Amirshahi SA, Koch M, Denzler J (2012) Phog-derived aesthetic measures applied to color photographs of artworks, natural scenes and objects Computer vision–ECCV 2012. Workshops and demonstrations. Springer, pp 522–531

  50. Rigau J, Feixas M, Sbert M (2008) Informational aesthetics measures. IEEE Comput Graph Appl 28(2):24–34

    Article  Google Scholar 

  51. Romero J, Machado P, Carballal A, Santos A (2012) Using complexity estimates in aesthetic image classification. J Math Arts 6(2-3):125–136

    Article  MathSciNet  Google Scholar 

  52. Saklofske D (1975) Visual aesthetic complexity, attractiveness and diversive exploration. Percept Mot Skills 41(3):813–814

    Article  Google Scholar 

  53. Schenkman BN, Jönsson FU (2000) Aesthetics and preferences of web pages. Behav Inform Technol 19(5):367–377

    Article  Google Scholar 

  54. Simond F, Arvanitopoulos Darginis N, Süsstrunk S (2015) Image aesthetics depends on context IEEE Proceedings of the international conference on image processing, EPFL-CONF-212967

    Google Scholar 

  55. Su HH, Chen TW, Kao CC, Hsu WH, Chien SY (2011) Scenic photo quality assessment with bag of aesthetics-preserving features Proceedings of the 19th ACM international conference on multimedia, pp 1213–1216

    Chapter  Google Scholar 

  56. Sun L, Yamasaki T, Aizawa K (2014) Relationship between visual complexity and aesthetics: Application to beauty prediction of photos Computer vision-ECCV 2014 workshops. Springer, pp 20–34

  57. Tian X, Dong Z, Yang K, Mei T (2015) Query-dependent aesthetic model with deep learning for photo quality assessment. IEEE Trans Multimedia 17 (11):2035–2048

    Article  Google Scholar 

  58. Tuch AN, Presslaber EE, StöCklin M, Opwis K, Bargas-Avila JA (2012) The role of visual complexity and prototypicality regarding first impression of websites: Working towards understanding aesthetic judgments. Int J Hum Comput Stud 70(11):794–811

    Article  Google Scholar 

  59. Van der Helm PA (2000) Simplicity versus likelihood in visual perception: from surprisals to precisals. Psychol Bull 126(5):770

  60. Vu CT, Phan TD, Chandler DM (2012) A spectral and spatial measure of local perceived sharpness in natural images. IEEE Trans Image Process 21(3):934–945

    Article  MathSciNet  MATH  Google Scholar 

  61. Wakefield KL, Baker J (1998) Excitement at the mall: determinants and effects on shopping response. J Retail 74(4):515–539

    Article  Google Scholar 

  62. Wang L, He DC (1990) Texture classification using texture spectrum. Pattern Recogn 23(8):905–910

    Article  Google Scholar 

  63. Xu Y, Ratcliff J, Scovell J, Speiginer G, Azuma R (2015) Real-time guidance camera interface to enhance photo aesthetic quality Proceedings of the 33rd annual ACM conference on human factors in computing systems, pp 1183–1186

    Google Scholar 

  64. Zhang L, Gao Y, Zimmermann R, Tian Q, Li X (2014) Fusion of multichannel local and global structural cues for photo aesthetics evaluation. IEEE Trans Image Process 23(3):1419–1429

    Article  MathSciNet  MATH  Google Scholar 

  65. Zhang Q, Shen X, Xu L, Jia J (2014) Rolling guidance filter Computer vision–ECCV 2014. Springer, pp 815–830

Download references

Author information

Authors and Affiliations

  1. Department of Information and Communication Engineering, The University of Tokyo, Faculty of Engineering Building 2, 7-3-1 Hongo, Bunkyo-ku, Tokyo, 113-8656, Japan

    Litian Sun, Toshihiko Yamasaki & Kiyoharu Aizawa

Authors
  1. Litian Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Toshihiko Yamasaki
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kiyoharu Aizawa
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Litian Sun.

Appendices

Appendix

ᅟ

A Parameter list

In the extraction of visual complexity features, we set parameters to constant values. We used arithmetic sequence or geometric sequence when setting the parameters to cover the parameter space as much as possible. Further feature selection is helpful to identify the optimal parameter settings. We list the value of these parameters in the following table (Table 6).

Table 6 Parameters to extract visual complexity features
Full size table

B Influence of classifier parameters

3.1 B.1 Subset with consented aesthetic scores

The aesthetics prediction task is implemented by a two-classes-classification method, in which an image is considered as of high quality if its public aesthetics rate is over a certain threshold. In such situation, it is necessary to check the variation among users’ aesthetics judgements to see how reliably the public aesthetics rate could reflect general users’ judgements. We annotate the rates for a certain photograph as \(R^{p_{i}}=\left \{r_{1},r{2},...r_{n} \right \}\), where p i refers to the ith sample in the dataset. We first assume that the distribution of aesthetics rates towards each photograph, follows a normal distribution. And we calculate the standard deviation of the aesthetics rates for each photograph. We consider the accuracy of human labelling as the ratio of the number of photographs, towards which most people agree with the aesthetics quality, in the dataset.

We set the consensus range as \(R_{avg}^{p_{i}}\pm \sigma ^{p_{i}} \), where \(R_{avg}^{p_{i}} \) is the public aesthetics rate, and \( \sigma ^{p_{i} } \) is the standard deviation for the sample. The two standard deviation gap leads to absolute majority (68%) of aesthetics quality judgements. If we divide the samples into two balanced classes, which means we take the average of the public rates for all the photographs in the dataset as the threshold. The threshold is defined as \(thres=avg(R_{avg}^{P})\), where P = p i|i ∈ (0, 255330) is the whole dataset. Photographs with the lower bondary of consensus range \(R_{avg}^{p_{i}}- \sigma ^{p_{i}}\) larger than the threshold, and the photographs with the higher boundary less than the threshold are considered to have converged aesthetics judgements.

In this way we find out that there are only 11198 photographs towards which absolute majority of the participants give a consensus aesthetics judgement, and the gap between the high and low quality of their public aesthetics rates is 1.91, with 6.47 for the lowest rate for high quality and 4.56 for the highest rate for low quality.

3.2 B.2 Aesthetic quality estimation with different classifier

We compare the aesthetic quality estimation accuracy obtained through different classifier with various parameter. As shown in Table 7, SVM with RBF kernel and the classification marginal set as 1000 has the best performance. Adjusting the estimator number of the AdaBoost classifier (base estimator as Decision Tree classifier) does not influence the accuracy very much and the performance is between that of linear SVM and RBF kernel SVM. Using AdaBoost classification, VCPC has the best performance when the number of estimator is set as 400, while VCHA achieve the best when the parameter is set as 800. Generally VCHA features have 1 2 % better accuracy than VCPC features.

Table 7 Comparison of the classification accuracy with different classifier
Full size table

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sun, L., Yamasaki, T. & Aizawa, K. Photo aesthetic quality estimation using visual complexity features. Multimed Tools Appl 77, 5189–5213 (2018). https://doi.org/10.1007/s11042-017-4424-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-017-4424-4

Keywords

Search

Navigation