Skip to main content
SpringerLink
Log in

Selective clustering for representative paintings selection

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Selective classification (or rejection based classification) has been proved useful in many applications. In this paper we describe a selective clustering framework with reject option to carry out large-scale digital arts analysis. With the help of deep learning techniques, we extract content-style features from a pre-trained convolutional network for the paintings. By proposing a rejection mechanism under Bayesian framework, we focus on selecting style-oriented representative paintings of an artist, which is an interesting and challenging cultural heritage application. Two kinds of samples are rejected during the rejection based robust continuous clustering process. Representative paintings are selected during the selective clustering phase. Visual qualitative analysis on small painting set and large scale quantitative experiments on a subset of Wikiart show that the proposed rejection based selective clustering approach outperforms the standard clustering methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Notes

  1. https://images.google.com/

References

  1. Arora RS, Elgammal A (2012) Towards automated classification of fine-art painting style: A comparative study. In: Proceedings of the 21st international conference on pattern recognition (ICPR), pp 3541– 3544

  2. Bartlett PL, Wegkamp MH (2008) Classification with a reject option using a hinge loss. J Mach Learn Res 9:1823–1840

    MathSciNet  MATH  Google Scholar 

  3. Chen CC, Amato G, Boujemaa N, Inria I, Pitas I, Alexander K, Kiernan K, Li CS, Wang ZJ (2002) Report of the delos-nsf working group on digital imagery for significant cultural and historical materials

  4. Chu WT, Wu YL (2016) Deep correlation features for image style classification. In: Proceedings of the 2016 ACM on multimedia conference, MM ’16. ACM, New York, pp 402–406

  5. Doersch C, Singh S, Gupta A, Sivic J, Efros AA (2012) What makes paris look like paris?. ACM Trans Graph 31(4):101:1–101:9

    Article  Google Scholar 

  6. Dumoulin V, Shlens J, Kudlur M (2017) A learned representation for artistic style. In: International conference on learning representations (ICLR). arXiv:1610.07629

  7. Dumoulin V, Shlens J, Kudlur M (2017) A learned representation for artistic style. In: ICLR, pp 9

  8. Fiedler M (1973) Algebraic connectivity of graphs. Czechoslov Math J 23(23):298–305

    MathSciNet  MATH  Google Scholar 

  9. Fumera G, Roli F, Giacinto G (2000) Reject option with multiple thresholds. Pattern Recogn 33(12):2099–2101

    Article  MATH  Google Scholar 

  10. Gatys LA, Ecker AS, Bethge M (2016) Image style transfer using convolutional neural networks. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 2414–2423

  11. Geifman Y, El-Yaniv R (2017) Selective classification for deep neural networks. In: Advances in neural information processing systems 30, pp 4878–4887. Curran Associates, Inc

  12. Golge E, Duygulu P (2015) FAME: Face association through model evolution. In: IEEE conference on computer vision and pattern recognition workshops (CVPRW), pp 43–49

  13. Grandvalet Y, Rakotomamonjy A, Keshet J, Canu S (2009) Support vector machines with a reject option. In: Advances in neural information processing systems 21, pp 537–544. Curran Associates, Inc

  14. Herbei R, Wegkamp MH (2006) Classification with reject option. Can J Stat 34(4):709–721

    Article  MathSciNet  MATH  Google Scholar 

  15. Hicsonmez S, Samet N, Sener F, Duygulu P (2017) DRAW: Deep networks for recognizing styles of artists who illustrate children’s books. In: Proceedings of the 2017 ACM on international conference on multimedia retrieval, ICMR’17. ACM, New York, pp 338–346

  16. Hu R, Li W, Kaick OV, Huang H, Averkiou M, Cohen-Or D, Zhang H (2017) Co-locating style-defining elements on 3d shapes. ACM Trans Graph 36(3):33:1–33:15

    Article  Google Scholar 

  17. Hughes JM, Graham DJ, Rockmore DN (2010) Quantification of artistic style through sparse coding analysis in the drawings of pieter bruegel the elder. Proc Natl Acad Sci 107(4):1279–1283

    Article  Google Scholar 

  18. Jain A, Gupta A, Rodriguez M, Davis LS (2013) Representing videos using mid-level discriminative patches. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 2571– 2578

  19. Jangtjik KA, Yeh MC, Hua KL (2016) Artist-based classification via deep learning with multi-scale weighted pooling. In: Proceedings of the 2016 ACM on Multimedia Conference, MM ’16. ACM, New York, pp 635–639

  20. Karmakar B, Pal NR (2018) How to make a neural network say “don’t know”. Inf Sci 430:444–466

    Article  MathSciNet  Google Scholar 

  21. Karypis G, Han EH, Kumar V (1999) Chameleon: Hierarchical clustering using dynamic modeling. IEEE computer society press

  22. Kim D, Son SW, Jeong H (2014) Large-scale quantitative analysis of painting arts. Sci Report 4:7370:1–7370:7

    Google Scholar 

  23. Li Q, Vempaty A, Varshney LR, Varshney PK (2017) Multi-object classification via crowdsourcing with a reject option. IEEE Trans Signal Process 65 (4):1068–1081

    Article  MathSciNet  MATH  Google Scholar 

  24. Li Y, Liu L, Shen C, Hengel AV (2017) Mining mid-level visual patterns with deep cnn activations. Int J Comput Vis 121(3):344–364

    Article  MathSciNet  Google Scholar 

  25. Liao J, Yao Y, Yuan L, Hua G, Kang SB (2017) Visual attribute transfer through deep image analogy. ACM Trans Graph 36(4):120:1–120:15

    Article  Google Scholar 

  26. Liu G, Yan Y, Ricci E, Yang Y, Han Y, Winkler S, Sebe N (2015) Inferring painting style with multi-task dictionary learning. In: Proceedings of the 24th international conference on artificial intelligence, IJCAI’15, pp 2162–2168. AAAI Press

  27. Ma D, Gao F, Bai Y, Lou Y, Wang S, Huang T, Duan L (2017) From part to whole: Who is behind the painting?. In: Proceedings of the 25rd ACM international conference on multimedia, pp 1174–1182. ACM

  28. Macqueen J (1965) Some methods for classification and analysis of multivariate observations. In: Proceedings of berkeley symposium on mathematical statistics and probability, pp 281–297

  29. Mao H, Cheung M, She J (2017) Deepart: Learning joint representation of visual arts. In: Proceedings of the 25rd ACM international conference on multimedia, pp 1183–1191. ACM

  30. Pudil P, Novovicova J, Blaha S, Kittler J (1992) Multistage pattern recognition with reject option. In: 11th IAPR international conference on pattern recognition, 1992. Vol. II. conference B: Pattern recognition methodology and systems, Proceedings, pp 92–95. IEEE

  31. Rodriguez A, Laio A (2014) Clustering by fast search and find of density peaks. Science 344(6191):1492–1496

    Article  Google Scholar 

  32. Sartori A, Culibrk D, Yan Y, Sebe N (2015) Who’s afraid of itten: Using the art theory of color combination to analyze emotions in abstract paintings. In: Proceedings of the 23rd ACM international conference on multimedia, MM’15. ACM, New York, pp 311–320

  33. Sartori A, Yan Y, Özbal G, Salah AAA, Salah AA, Sebe N (2015) Looking at mondrian’s victory boogie-woogie: What do i feel?. In: Proceedings of the 24th international conference on artificial intelligence, IJCAI’15, pp 2503–2509. AAAI Press

  34. Shah SA, Koltun V (2017) Robust continuous clustering. Proc Natl Acad Sci 114(37):9814–9819

    Article  Google Scholar 

  35. Shamir L, Macura T, Orlov N, Eckley DM, Goldberg IG (2010) Impressionism, expressionism, surrealism: Automated recognition of painters and schools of art. ACM Trans Appl Percept 7(2):8:1– 8:17

    Article  Google Scholar 

  36. Shen J (2009) Stochastic modeling western paintings for effective classification. Pattern Recogn 42(2):293–301

    Article  MATH  Google Scholar 

  37. Soltanolkotabi M, Elhamifar E, Candès EJ (2013) Robust subspace clustering. Ann Statist 42(2):669–699

    Article  MathSciNet  MATH  Google Scholar 

  38. Srivastava A, Zou J, Sutton C (2016) Clustering with a reject option: Interactive clustering as bayesian prior elicitation. In: ICML workshop on human interpretability in machinelearning(WHI). New York, NY, USA

  39. Tax DM, Duin RP (2008) Growing a multi-class classifier with a reject option. Pattern Recogn Lett 29(10):1565–1570

    Article  Google Scholar 

  40. Taylor RP, Micolich AP, Jonas D (1999) Fractal analysis of pollock’s drip paintings. Nature 399(6735): 422–422

    Article  Google Scholar 

  41. van Noord N, Hendriks E, Postma E (2015) Toward discovery of the artist’s style: Learning to recognize artists by their artworks. IEEE Signal Process Mag 32(4):46–54

    Article  Google Scholar 

  42. Wang J, Yang J, Yu K, Lv F, Huang T, Gong Y (2010) Locality-constrained linear coding for image classification. In: Computer vision and pattern recognition, pp 3360–3367

  43. Wikiart.org - visual art encyclopedia. https://www.wikiart.org//. Accessed January 30, 2018

  44. Zhang K, Harrell S, Ji X (2012) Computational aesthetics: on the complexity of computer-generated paintings. Leonardo 45(2):243–248

    Article  Google Scholar 

  45. Zhang X, Hu BG (2012) Learning in the class imbalance problem when costs are unknown for errors and rejects. In: 2012 IEEE 12th international conference on data mining workshops (ICDMW), pp 194–201. IEEE

Download references

Acknowledgements

This work was supported by National Natural Science Foundation of China under nos. 61832016, 61672520 and 61702488, as well as the independent research project of National Laboratory of Pattern Recognition.

Author information

Authors and Affiliations

  1. NLPR-LIAMA, Institute of Automation, Chinese Academy of Sciences, Beijing, China

    Yingying Deng, Fan Tang, Weiming Dong & Changsheng Xu

  2. University of Chinese Academy of Sciences, Beijing, China

    Yingying Deng & Fan Tang

  3. Institute of Software, Chinese Academy of Sciences, Beijing, China

    Fuzhang Wu

  4. University of Konstanz, Konstanz, Germany

    Oliver Deussen

Authors
  1. Yingying Deng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Fan Tang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Weiming Dong
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Fuzhang Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Oliver Deussen
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Changsheng Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Weiming Dong.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Deng, Y., Tang, F., Dong, W. et al. Selective clustering for representative paintings selection. Multimed Tools Appl 78, 19305–19323 (2019). https://doi.org/10.1007/s11042-019-7271-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-019-7271-7

Keywords

Search

Navigation