Skip to main content
SpringerLink
Log in

Art painting detection and identification based on deep learning and image local features

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Many art paintings are placed in film scenes or TV programs as decoration. To prevent using unauthorized copyrighted art paintings, we propose a method that combines a deep learning based object detector and hand-crafted image local features to identify copyrighted art paintings from images that contain them. The object detector is trained with our collected data to be able to detect art paintings. If a query image is input, the object detector will detect the art painting regions, then, the copyrighted art paintings can be identified by matching image local features between the art painting regions and the original copyrighted art paintings that have already been stored in advance. To test the ability of the proposed method from different aspects, we prepared four different kinds of test images: Famous, Monitor Easy, Monitor Hard, and Print. Finally, we provide a practicability analysis of our method based on the experimental results on these test images. Additionally, compared with Scale Invariant Feature Transform (SIFT), our approach outperformed by more than 20%.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Notes

  1. http://cclabs.smu.ac.kr/research/datasets-for-art-painting-detection-and-identification/

  2. https://pjreddie.com/darknet/yolo/

  3. https://github.com/rghunter/BRISK

  4. http://www.cs.ubc.ca/~lowe/keypoints/

  5. http://www.ipol.im/pub/art/2011/my-asift/

References

  1. Alahi A, Ortiz R, Vandergheynst P (2012) Freak: fast retina keypoint. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. doi:https://doi.org/10.1109/CVPR.2012.6247715

  2. Bay H, Tuytelaars T, Van Gool L. (2006) Surf: Speeded up robust features. In ECCV. doi:https://doi.org/10.1007/11744023_32

    Chapter  Google Scholar 

  3. Bosch A, Zisserman A, Munoz X (2007) Image classification using random forests and ferns. ICCV, pp. 1–8

  4. Brown M, Lowe DG (2007) Automatic panoramic image stitching using invariant features. In. Int J Comput Vis 74:59–77. https://doi.org/10.1007/s11263-006-0002-3

    Article  Google Scholar 

  5. Cui J, Liu Y, Xu Y, Zhao H, Zha H (2013) Tracking generic human motion via fusion of low- and high-dimensional approaches. IEEE Trans Syst Man Cybern Syst 43(4):996–1002

    Article  Google Scholar 

  6. Dai J, Li Y, He K, Sun J. R-FCN: Object Detection via Region-based Fully Convolutional Networks. arXiv preprint arXiv:1605.06409

  7. Deac A, van der Lubbe J, Backer E (2006) Feature selection for paintings classification by optimal tree pruning. in Multimedia Content Representation, Classification and Security. pp. 354–361

    Chapter  Google Scholar 

  8. Deng J, Dong W, Socher R, Li L-J, Li K, Fei-Fei L (2009) ImageNet: A Large-Scale Hierarchical Image Database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. doi:https://doi.org/10.1109/CVPR.2009.5206848

  9. Everingham M, Gool LV, Williams C, Winn J, Zisserman A (2010) The Pascal visual object classes (VOC). Challenge 88:303–338. https://doi.org/10.1007/s11263-009-0275-4 Available online: http://host.robots.ox.ac.uk/pascal/VOC/ (accessed on 13.09.2017)

    Article  Google Scholar 

  10. Famous Artworks Exhibition. Available online: http://www.ibiblio.org/wm/paint/ (accessed on 13.09.2017)

  11. Food Dataset. Available online: http://foodcam.mobi/dataset.html (accessed on 13.09.2017)

  12. Girshick R (2015) Fast R-CNN. In International Conference on Computer Vision. doi:https://doi.org/10.1109/ICCV.2015.169

  13. Girshick R, Donahue J, Darrell T, Malik J (2014) Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. doi:https://doi.org/10.1109/CVPR.2014.81

  14. Jain AK, Duin RPW, Mao J (2000) Statistical pattern recognition: a review. IEEE Trans Pattern Anal Mach Intell 22:4–37. https://doi.org/10.1109/34.824819

    Article  Google Scholar 

  15. Keren D (2002) Painter identification using local features and naive bayes. in Pattern Recognition. 2002. Proceedings. 16th International Conference on. vol. 2, pp. 474–477

  16. Kim K.-H, Hong S, Roh B, Cheon Y, Park M. Pvanet: Deep but lightweight neural networks for real-time object detection. arXiv preprint arXiv:1608.08021

  17. Leutenegger S, Chli M, Siewart R. (2011) Brisk: Binary robust invariant scalable keypoints. In International Conference on Computer Vision. doi:https://doi.org/10.1109/ICCV.2011.6126542

  18. Li J, Wang J (2004) Studying digital imagery of ancient paintings by mixtures of stochastic models. Image Processing. IEEE Transactions on 13(3):340–353

    Google Scholar 

  19. Lin T-Y, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Dollr P, Zitnick CL (2014) Microsoft COCO: Common objects in context. In ECCV. doi:https://doi.org/10.1007/978-3-319-10602-1_48

    Google Scholar 

  20. Lin T-Y, Dollar P, Girshick R, He K, Hariharan B, Belongie S (2017) Feature pyramid networks for object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. doi:https://doi.org/10.1109/TPAMI.2014.2300479

    Article  Google Scholar 

  21. Liu W, Anguelov D, Erhan D, Szegedy C, Reed S, Fu C-Y, Berg AC (2016) SSD: Single Shot Multibox Detector. In ECCV. doi:https://doi.org/10.1007/978-3-319-46448-0_2

    Chapter  Google Scholar 

  22. Lombardi T, Cha S-H, Tappert C (2004) A graphical user interface for a fine-art painting image retrieval system. in MIR ‘04: Proceedings of the 6th ACM SIGMM international workshop on Multimedia information retrieval. ACM. pp. 107–112

  23. Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vis 60:91–110. https://doi.org/10.1023/B:VISI.0000029664.99615.94

    Article  Google Scholar 

  24. Mair E, Hager GD, Burschka D, Suppa M, Hirzinger G (2010) Adaptive and generic corner detection based on the accelerated segment test. In ECCV. doi:https://doi.org/10.1007/978-3-642-15552-9_14

    Chapter  Google Scholar 

  25. Martinel N, Micheloni C, Foresti GL (2013) Robust painting recognition and registration for mobile augmented reality. IEEE Signal Process Letter 20(11):1022–1025

    Article  Google Scholar 

  26. Mikolajczyk K, Schmid C (2005) A performance evaluation of local descriptors. IEEE Trans Pattern Anal Mach Intell 27:1615–1630. https://doi.org/10.1109/TPAMI.2005.188

    Article  Google Scholar 

  27. Miksik O, Mikolajczyk K (2012) Evaluation of local detectors and descriptors for fast feature matching. In International Conference on Pattern Recognition

  28. Morel KM, Yu G (2009) ASIFT: a new framework for fully affine invariant image comparison. SIAM Journal on Imaging Sciences 2:438–469. https://doi.org/10.1137/080732730

    Article  MathSciNet  MATH  Google Scholar 

  29. Redmon J, Farhadi A. Yolo9000: Better, faster, stronger. arXiv preprint arXiv:1612.08242

  30. Redmon J, Divvala S, Girshick R, Farhadi A (2016) You only look once: unified, real-time object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. doi:https://doi.org/10.1109/CVPR.2016.91

  31. Ren S, He K, Girshick R, Sun J (2017) Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans Pattern Anal Mach Intell 39:1137–1149. https://doi.org/10.1109/TPAMI.2016.2577031

    Article  Google Scholar 

  32. Ringgold v. Black Entm’t Television (1997) Inc., 126 F.3d 70 (2d Cir)

  33. Rublee E, Rabaud V, Konolige K, Bradski G (2011) ORB: an efficient alternative to SIFT or SURF. In International Conference on Computer Vision. doi:https://doi.org/10.1109/ICCV.2011.6126544

  34. Shrivastava A, Gupta A (2016) Contextual Priming and Feedback for Faster R-CNN. In ECCV, 330–348. doi:https://doi.org/10.1007/978-3-319-46448-0_20

    Chapter  Google Scholar 

  35. Shrivastava A, Gupta A, Girshick R. Training Region-based Object Detectors with Online Hard Example Mining. arXiv preprint arXiv:1604.03540

  36. Skrypnyk I, Lowe DG (2004) Scene Modelling, Recognition and Tracking with Invariant Image Features. In International symposium on mixed and augmented reality. doi:https://doi.org/10.1109/ISMAR.2004.53

  37. Szeliski R (2006) Image alignment and stitching: a tutorial. Foundations and Trends in Computer Graphics and Vision 2:1–104. https://doi.org/10.1561/0600000009

    Article  MathSciNet  MATH  Google Scholar 

  38. Uijlings JRR, van de Sande KEA, Gevers T, Smeulders AWM (2013) Selective search for object recognition. Int J Comput Vis 104:154–171. https://doi.org/10.1007/s11263-013-0620-5

    Article  Google Scholar 

  39. Wikiart. Available online: https://www.wikiart.org/en/paintings-by-genre (accessed on 13.09.2017)

  40. Xu L, Oja E (1993) Randomized Hough transform (RHT): basic mechanisms, algorithms, and computational complexities. CVGIP: Image Understand 57(2):131–154

    Article  Google Scholar 

  41. Yang B, Yan J, Lei Z, Li SZ (2016) Craft Objects from Images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. doi:https://doi.org/10.1109/CVPR.2016.650

  42. Zagoruyko S, Lerer A, Lin T-Y, Pinheiro PO, Gross S, Chintala S, Dollar P (2016) A multipath network for object detection. In BMVC, doi:https://doi.org/10.5244/C.30.15

  43. Zhai A, Kislyuk D, Jing Y, Feng M, Tzeng E, Donahue J, Du YL, Darrell T (2017) Visual discovery at Pinterest. In Proceedings of the 26th International Conference on World Wide Web Companion. doi:https://doi.org/10.1145/3041021.3054201

Download references

Acknowledgments

This research is supported by Ministry of Culture, Sports and Tourism(MCST) and Korea Creative Content Agency(KOCCA) in the Culture Technology (CT) Research & Development Program 2017.

Author information

Authors and Affiliations

  1. Department of Copyright Protection, Sangmyung University, Seoul, South Korea

    Yiyu Hong

  2. Department of Electronics Engineering, Sangmyung, University, Seoul, South Korea

    Jongweon Kim

Authors
  1. Yiyu Hong
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jongweon Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jongweon Kim.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hong, Y., Kim, J. Art painting detection and identification based on deep learning and image local features. Multimed Tools Appl 78, 6513–6528 (2019). https://doi.org/10.1007/s11042-018-6387-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-018-6387-5

Keywords

Search

Navigation