Multimedia Tools and Applications

, Volume 78, Issue 14, pp 19305–19323 | Cite as

Selective clustering for representative paintings selection

  • Yingying Deng
  • Fan Tang
  • Weiming DongEmail author
  • Fuzhang Wu
  • Oliver Deussen
  • Changsheng Xu


Selective classification (or rejection based classification) has been proved useful in many applications. In this paper we describe a selective clustering framework with reject option to carry out large-scale digital arts analysis. With the help of deep learning techniques, we extract content-style features from a pre-trained convolutional network for the paintings. By proposing a rejection mechanism under Bayesian framework, we focus on selecting style-oriented representative paintings of an artist, which is an interesting and challenging cultural heritage application. Two kinds of samples are rejected during the rejection based robust continuous clustering process. Representative paintings are selected during the selective clustering phase. Visual qualitative analysis on small painting set and large scale quantitative experiments on a subset of Wikiart show that the proposed rejection based selective clustering approach outperforms the standard clustering methods.


Digital arts analysis Pattern mining Rejection mechanism Deep feature representation 



This work was supported by National Natural Science Foundation of China under nos. 61832016, 61672520 and 61702488, as well as the independent research project of National Laboratory of Pattern Recognition.


  1. 1.
    Arora RS, Elgammal A (2012) Towards automated classification of fine-art painting style: A comparative study. In: Proceedings of the 21st international conference on pattern recognition (ICPR), pp 3541– 3544Google Scholar
  2. 2.
    Bartlett PL, Wegkamp MH (2008) Classification with a reject option using a hinge loss. J Mach Learn Res 9:1823–1840MathSciNetzbMATHGoogle Scholar
  3. 3.
    Chen CC, Amato G, Boujemaa N, Inria I, Pitas I, Alexander K, Kiernan K, Li CS, Wang ZJ (2002) Report of the delos-nsf working group on digital imagery for significant cultural and historical materialsGoogle Scholar
  4. 4.
    Chu WT, Wu YL (2016) Deep correlation features for image style classification. In: Proceedings of the 2016 ACM on multimedia conference, MM ’16. ACM, New York, pp 402–406Google Scholar
  5. 5.
    Doersch C, Singh S, Gupta A, Sivic J, Efros AA (2012) What makes paris look like paris?. ACM Trans Graph 31(4):101:1–101:9CrossRefGoogle Scholar
  6. 6.
    Dumoulin V, Shlens J, Kudlur M (2017) A learned representation for artistic style. In: International conference on learning representations (ICLR). arXiv:1610.07629
  7. 7.
    Dumoulin V, Shlens J, Kudlur M (2017) A learned representation for artistic style. In: ICLR, pp 9Google Scholar
  8. 8.
    Fiedler M (1973) Algebraic connectivity of graphs. Czechoslov Math J 23(23):298–305MathSciNetzbMATHGoogle Scholar
  9. 9.
    Fumera G, Roli F, Giacinto G (2000) Reject option with multiple thresholds. Pattern Recogn 33(12):2099–2101CrossRefzbMATHGoogle Scholar
  10. 10.
    Gatys LA, Ecker AS, Bethge M (2016) Image style transfer using convolutional neural networks. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 2414–2423Google Scholar
  11. 11.
    Geifman Y, El-Yaniv R (2017) Selective classification for deep neural networks. In: Advances in neural information processing systems 30, pp 4878–4887. Curran Associates, IncGoogle Scholar
  12. 12.
    Golge E, Duygulu P (2015) FAME: Face association through model evolution. In: IEEE conference on computer vision and pattern recognition workshops (CVPRW), pp 43–49Google Scholar
  13. 13.
    Grandvalet Y, Rakotomamonjy A, Keshet J, Canu S (2009) Support vector machines with a reject option. In: Advances in neural information processing systems 21, pp 537–544. Curran Associates, IncGoogle Scholar
  14. 14.
    Herbei R, Wegkamp MH (2006) Classification with reject option. Can J Stat 34(4):709–721MathSciNetCrossRefzbMATHGoogle Scholar
  15. 15.
    Hicsonmez S, Samet N, Sener F, Duygulu P (2017) DRAW: Deep networks for recognizing styles of artists who illustrate children’s books. In: Proceedings of the 2017 ACM on international conference on multimedia retrieval, ICMR’17. ACM, New York, pp 338–346Google Scholar
  16. 16.
    Hu R, Li W, Kaick OV, Huang H, Averkiou M, Cohen-Or D, Zhang H (2017) Co-locating style-defining elements on 3d shapes. ACM Trans Graph 36(3):33:1–33:15CrossRefGoogle Scholar
  17. 17.
    Hughes JM, Graham DJ, Rockmore DN (2010) Quantification of artistic style through sparse coding analysis in the drawings of pieter bruegel the elder. Proc Natl Acad Sci 107(4):1279–1283CrossRefGoogle Scholar
  18. 18.
    Jain A, Gupta A, Rodriguez M, Davis LS (2013) Representing videos using mid-level discriminative patches. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 2571– 2578Google Scholar
  19. 19.
    Jangtjik KA, Yeh MC, Hua KL (2016) Artist-based classification via deep learning with multi-scale weighted pooling. In: Proceedings of the 2016 ACM on Multimedia Conference, MM ’16. ACM, New York, pp 635–639Google Scholar
  20. 20.
    Karmakar B, Pal NR (2018) How to make a neural network say “don’t know”. Inf Sci 430:444–466MathSciNetCrossRefGoogle Scholar
  21. 21.
    Karypis G, Han EH, Kumar V (1999) Chameleon: Hierarchical clustering using dynamic modeling. IEEE computer society pressGoogle Scholar
  22. 22.
    Kim D, Son SW, Jeong H (2014) Large-scale quantitative analysis of painting arts. Sci Report 4:7370:1–7370:7Google Scholar
  23. 23.
    Li Q, Vempaty A, Varshney LR, Varshney PK (2017) Multi-object classification via crowdsourcing with a reject option. IEEE Trans Signal Process 65 (4):1068–1081MathSciNetCrossRefzbMATHGoogle Scholar
  24. 24.
    Li Y, Liu L, Shen C, Hengel AV (2017) Mining mid-level visual patterns with deep cnn activations. Int J Comput Vis 121(3):344–364MathSciNetCrossRefGoogle Scholar
  25. 25.
    Liao J, Yao Y, Yuan L, Hua G, Kang SB (2017) Visual attribute transfer through deep image analogy. ACM Trans Graph 36(4):120:1–120:15CrossRefGoogle Scholar
  26. 26.
    Liu G, Yan Y, Ricci E, Yang Y, Han Y, Winkler S, Sebe N (2015) Inferring painting style with multi-task dictionary learning. In: Proceedings of the 24th international conference on artificial intelligence, IJCAI’15, pp 2162–2168. AAAI PressGoogle Scholar
  27. 27.
    Ma D, Gao F, Bai Y, Lou Y, Wang S, Huang T, Duan L (2017) From part to whole: Who is behind the painting?. In: Proceedings of the 25rd ACM international conference on multimedia, pp 1174–1182. ACMGoogle Scholar
  28. 28.
    Macqueen J (1965) Some methods for classification and analysis of multivariate observations. In: Proceedings of berkeley symposium on mathematical statistics and probability, pp 281–297Google Scholar
  29. 29.
    Mao H, Cheung M, She J (2017) Deepart: Learning joint representation of visual arts. In: Proceedings of the 25rd ACM international conference on multimedia, pp 1183–1191. ACMGoogle Scholar
  30. 30.
    Pudil P, Novovicova J, Blaha S, Kittler J (1992) Multistage pattern recognition with reject option. In: 11th IAPR international conference on pattern recognition, 1992. Vol. II. conference B: Pattern recognition methodology and systems, Proceedings, pp 92–95. IEEEGoogle Scholar
  31. 31.
    Rodriguez A, Laio A (2014) Clustering by fast search and find of density peaks. Science 344(6191):1492–1496CrossRefGoogle Scholar
  32. 32.
    Sartori A, Culibrk D, Yan Y, Sebe N (2015) Who’s afraid of itten: Using the art theory of color combination to analyze emotions in abstract paintings. In: Proceedings of the 23rd ACM international conference on multimedia, MM’15. ACM, New York, pp 311–320Google Scholar
  33. 33.
    Sartori A, Yan Y, Özbal G, Salah AAA, Salah AA, Sebe N (2015) Looking at mondrian’s victory boogie-woogie: What do i feel?. In: Proceedings of the 24th international conference on artificial intelligence, IJCAI’15, pp 2503–2509. AAAI PressGoogle Scholar
  34. 34.
    Shah SA, Koltun V (2017) Robust continuous clustering. Proc Natl Acad Sci 114(37):9814–9819CrossRefGoogle Scholar
  35. 35.
    Shamir L, Macura T, Orlov N, Eckley DM, Goldberg IG (2010) Impressionism, expressionism, surrealism: Automated recognition of painters and schools of art. ACM Trans Appl Percept 7(2):8:1– 8:17CrossRefGoogle Scholar
  36. 36.
    Shen J (2009) Stochastic modeling western paintings for effective classification. Pattern Recogn 42(2):293–301CrossRefzbMATHGoogle Scholar
  37. 37.
    Soltanolkotabi M, Elhamifar E, Candès EJ (2013) Robust subspace clustering. Ann Statist 42(2):669–699MathSciNetCrossRefzbMATHGoogle Scholar
  38. 38.
    Srivastava A, Zou J, Sutton C (2016) Clustering with a reject option: Interactive clustering as bayesian prior elicitation. In: ICML workshop on human interpretability in machinelearning(WHI). New York, NY, USAGoogle Scholar
  39. 39.
    Tax DM, Duin RP (2008) Growing a multi-class classifier with a reject option. Pattern Recogn Lett 29(10):1565–1570CrossRefGoogle Scholar
  40. 40.
    Taylor RP, Micolich AP, Jonas D (1999) Fractal analysis of pollock’s drip paintings. Nature 399(6735): 422–422CrossRefGoogle Scholar
  41. 41.
    van Noord N, Hendriks E, Postma E (2015) Toward discovery of the artist’s style: Learning to recognize artists by their artworks. IEEE Signal Process Mag 32(4):46–54CrossRefGoogle Scholar
  42. 42.
    Wang J, Yang J, Yu K, Lv F, Huang T, Gong Y (2010) Locality-constrained linear coding for image classification. In: Computer vision and pattern recognition, pp 3360–3367Google Scholar
  43. 43. - visual art encyclopedia. Accessed January 30, 2018
  44. 44.
    Zhang K, Harrell S, Ji X (2012) Computational aesthetics: on the complexity of computer-generated paintings. Leonardo 45(2):243–248CrossRefGoogle Scholar
  45. 45.
    Zhang X, Hu BG (2012) Learning in the class imbalance problem when costs are unknown for errors and rejects. In: 2012 IEEE 12th international conference on data mining workshops (ICDMW), pp 194–201. IEEEGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.NLPR-LIAMA, Institute of AutomationChinese Academy of SciencesBeijingChina
  2. 2.University of Chinese Academy of SciencesBeijingChina
  3. 3.Institute of SoftwareChinese Academy of SciencesBeijingChina
  4. 4.University of KonstanzKonstanzGermany

Personalised recommendations