Skip to main content
SpringerLink
Log in

Automatic portrait oil painter: joint domain stylization for portrait images

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Everyone has the dream of being in the center of famous art paintings, admired by numerous future generations. However, the dream came true at a huge cost of the painter’s commission in old days. In our paper, another practical choice is provided for everyone to achieve that dream – an automatic portrait oil painter transferring some artistic styles from one single reference painting. To address this issue, we propose a joint-domain image stylization approach, particularly for portrait oil paintings. From the view of artistic appreciation, we analyze an amount of oil painting art works and summarize three critical factors to depict the figure, i.e. color, structure and texture. Based on this point, we separate and represent an artistic work into these three domains. Then, considering their intrinsic properties and following an art creation route, we propose the corresponding approaches to jointly model and transfer the features in these domains. First, a swatch-based color adjustment is proposed to recolor the tone of the input image based on semantic regions corresponding to the references. Second, the main structures of the input image is maintained by sparse reconstruction. Third, a coarse-to-fine texture synthesis is used to enhance the detail oil painting patterns. Extensive experimental results demonstrate that the proposed method achieves desirable results compared with state-of-the-art methods in not only transferring the styles from references but also keeping consistent contents with the given portrait.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Ashikhmin M (2003) Fast texture transfer. IEEE Comput Graph Appl 23(4):38–43

    Article  Google Scholar 

  2. Berger I, Shamir A, Mahler M, Carter E, Hodgins J (2013) Style and abstraction in portrait sketching. ACM Trans Graph 32(4):96–96

    Article  Google Scholar 

  3. Bhujle H, Chaudhuri S (2014) Novel speed-up strategies for non-local means denoising with patch and edge patch based dictionaries. IEEE Trans Image Process 23(1):356–365

    Article  MathSciNet  MATH  Google Scholar 

  4. Bousseau A, Kaplan M, Thollot J, Sillion F (1983) Interactive watercolor rendering with temporal coherence and abstraction. In: International Symposium on Non-photorealistic Animation and Rendering, pp 141–149

  5. Curtis CJ, Anderson SE, Seims JE, Fleischer KW, Salesin D (1997) Computer-generated watercolor. In: Conference on Computer Graphics and Interactive Techniques, pp 421–430

  6. Dezeustre G (2014) Glaze app https://itunes.apple.com/us/app/glaze/id521573656

  7. Donoho DL (2006) Compressed sensing. IEEE Trans Inf Theory 52(4):1289–1306

    Article  MathSciNet  MATH  Google Scholar 

  8. Efros A, Leung TK et al (1999) Texture synthesis by non-parametric sampling. In: Proceedings of the seventh IEEE International Conference on Computer Vision, vol 2. IEEE, pp 1033–1038

  9. Efros AA, Freeman WT (2001) Image quilting for texture synthesis and transfer. In: Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques. ACM, pp 341–346

  10. Flanagan P, Cavanagh P, Favreau OE (1990) Independent orientation-selective mechanisms for the cardinal directions of colour space. Vis Res 30(5):769–778

    Article  Google Scholar 

  11. Frigo O, Sabater N, Delon J, Hellier P (2016) Split and match: example-based adaptive patch sampling for unsupervised style transfer. In: IEEE Conference on Computer Vision and Pattern Recognition, pp 553–561

  12. Gatys LA, Ecker AS, Bethge M (2016) Image style transfer using convolutional neural networks. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

  13. Greenfield GR, House DH (2003) Image recoloring induced by palette color associations. Journal of Wscg 11:189–196

    Google Scholar 

  14. He K, Sun J (2014) Image completion approaches using the statistics of similar patches. IEEE Trans Pattern Anal Mach Intell 36(12):2423–2435

    Article  Google Scholar 

  15. Hertzmann A (1998) Painterly rendering with curved brush strokes of multiple sizes. In: Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques. ACM, pp 453–460

  16. Hertzmann A, Jacobs CE, Oliver N, Curless B, Salesin D (2001) Image analogies. In: Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques. ACM, pp 327–340

  17. Huang TW, Chen HT (2009) Landmark-based sparse color representations for color transfer. In: IEEE 12Th International Conference on Computer Vision. IEEE, pp 199–204

  18. Jia K, Wang X, Tang X (2013) Image transformation based on learning dictionaries across image spaces. IEEE Trans Pattern Anal Mach Intell 35(2):367–380

    Article  Google Scholar 

  19. Kyprianidis JE, Collomosse J, Wang T, Isenberg T (2013) State of the art: a taxonomy of artistic stylization techniques for images and video. IEEE Trans Vis Comput Graph 19(5):866–885

    Article  Google Scholar 

  20. Lee H, Seo S, Ryoo S, Yoon K (2010) Directional texture transfer. In: Proceedings of the 8th International Symposium on Non-Photorealistic Animation and Rendering. ACM, pp 43–48

  21. Levin A, Lischinski D, Weiss Y (2004) Colorization using optimization. ACM Trans Graph 23(3):689–694

    Article  Google Scholar 

  22. Levin A, Lischinski D, Weiss Y (2008) A closed-form solution to natural image matting. IEEE Trans Pattern Anal Mach Intell 30(2):228–242

    Article  Google Scholar 

  23. Li C, Wand M (2016) Combining markov random fields and convolutional neural networks for image synthesis. In: IEEE Conference on Computer Vision and Pattern Recognition, pp 2479–2486

  24. Liu Y, Zheng Y, Liang Y, Liu S, Rosenblum DS (2016) Urban water quality prediction based on multi-task multi-view learning. In: Proceedings of the International Joint Conference on Artificial Intelligence

  25. Liu X, Ma L, Liu Y (2017) Global tone: using tone to draw in pen-and-ink illustration. Multimed Tool Appl 76(10):12853–12869

    Article  Google Scholar 

  26. Lo KH, Wang YCF, Hua KL (2016) Example-based image textural style transfer. IEEE Multimed 23(4):60–66

    Article  Google Scholar 

  27. Lu Y, Wei Y, Liu L, Zhong J, Sun L, Liu Y (2017) Towards unsupervised physical activity recognition using smartphone accelerometers. Multimed Tool Appl 76(8):10,701–10,719

    Article  Google Scholar 

  28. Marr D (1982) Vision: a computational investigation into the human representation and processing of visual information. Henry Holt and Co., Inc., New York

    Google Scholar 

  29. Meitu app. https://itunes.apple.com/cn/app/id463422433

  30. Paget R, Longstaff D (1995) Texture synthesis via a non-parametric markov random field. In: Proceedings of DICTA-95, Digital Image Computing: Techniques and Applications, vol 1, pp 547–552

  31. Pérez P, Gangnet M, Blake A (2003) Poisson image editing. ACM Trans Graph 22(3):313–318

    Article  Google Scholar 

  32. Porter T, Duff T (1984) Compositing digital images. ACM Siggraph Comput Graph 18(3):253–259

    Article  Google Scholar 

  33. Protter M, Elad M (2009) Image sequence denoising via sparse and redundant representations. IEEE Trans Image Process 18(1):27–35

    Article  MathSciNet  MATH  Google Scholar 

  34. Reinhard E, Ashikhmin M, Gooch B, Shirley P (2001) Color transfer between images. IEEE Comput Graph Appl 21(5):34–41

    Article  Google Scholar 

  35. Rosales R, Achan K, Frey B (2003) Unsupervised image translation. In: Proceedings of IEEE International Conference on Computer Vision, pp 472–472

  36. Rother C, Kolmogorov V, Blake A (2004) Grabcut: interactive foreground extraction using iterated graph cuts. ACM Trans Graph 23(3):309–314

    Article  Google Scholar 

  37. Ruderman DL, Cronin TW, Chiao CC (1998) Statistics of cone responses to natural images: implications for visual coding. J Opt Soc Am 15(8):2036–2045

    Article  Google Scholar 

  38. Salisbury M, Anderson CR, Lischinski D, Salesin D (1996) Scale-dependent reproduction of pen-and-ink illustrations. Br J Sports Med 48(7):622

    Google Scholar 

  39. Saragih JM, Lucey S, Cohn JF (2009) Face alignment through subspace constrained mean-shifts. In: IEEE 12th International Conference on Computer Vision. IEEE, pp 1034–1041

  40. Shih YC, Paris S, Barnes C, Freeman WT, Durand F (2014) Style transfer for headshot portraits. ACM Trans Graph 33(4):1–14

    Article  Google Scholar 

  41. Sunkavalli K, Johnson MK, Matusik W, Pfister H (2010) Multi-scale image harmonization. ACM Trans Graph 29(4):125

    Article  Google Scholar 

  42. Wang X, Tang X (2009) Face photo-sketch synthesis and recognition. IEEE Trans Pattern Anal Mach Intell 31(11):1955–1967

    Article  MathSciNet  Google Scholar 

  43. Wang S, Zhang L, Liang Y, Pan Q (2012) Semi-coupled dictionary learning with applications to image super-resolution and photo-sketch synthesis. In: IEEE Conference on Computer Vision and Pattern Recognition. IEEE, pp 2216–2223

  44. Wang T, Collomosse J, Hunter A, Greig D (2013) Learnable stroke models for example-based portrait painting. In: British Machine Vision Conference, pp 36.1–36.11

  45. Wang Q, Chen D, Li S, Wu Q, Zhang Q (2017) An adaptive cartoon-like stylization for color video in real time. Multimed Tool Appl 76(15):16767–16782

    Article  Google Scholar 

  46. Welsh T, Ashikhmin M, Mueller K (2002) Transferring color to greyscale images. ACM Trans Graph 21(3):277–280

    Article  Google Scholar 

  47. Winkenbach GA, Salesin D (1994) Computer-generated pen-and-ink illustration. In: Conference on Computer Graphics and Interactive Techniques, pp 91–100

  48. Yang J, Wright J, Huang TS, Ma Y (2010) Image super-resolution via sparse representation. IEEE Trans Image Process 19(11):2861–2873

    Article  MathSciNet  MATH  Google Scholar 

  49. Yang L, Chen WB, Zhang C, Johnstone JK, Gao S, Warner G (2012) Profiling online auction sellers using image-editing styles. IEEE Multimed 19(1):29–29

    Article  Google Scholar 

  50. Yang Y, Zhao H, You L, Tu R, Wu X, Jin X (2017) Semantic portrait color transfer with internet images. Multimed Tool Appl 76:523–541

    Article  Google Scholar 

  51. Zhang W, Cao C, Chen S, Liu J (2013) Style transfer via image component analysis. IEEE Trans Multimed 15(7):1594–1601

    Article  Google Scholar 

  52. Zhao M, Zhu SC (2010) Sisley the abstract painter. In: Proceedings of the 8th International Symposium on Non-Photorealistic Animation and Rendering. ACM, pp 99–107

  53. Zhao M, Zhu SC (2011) Portrait painting using active templates. In: International Symposium on Non-Photorealistic Animation and Rendering 2009, Vancouver, Bc, Canada, August, pp 117–124

  54. Zhao M, Zhu SC (2013) Abstract painting with interactive control of perceptual entropy. ACM Transactions on Applied Perception 10(1):5

    Article  Google Scholar 

Download references

Acknowledgments

This work was supported by the National Natural Science Foundation of China under Contract 61472011.

Author information

Authors and Affiliations

  1. Institute of Computer Science and Technology, Peking University, Beijing, China

    Saboya Yang, Shuai Yang, Wenhan Yang & Jiaying Liu

Authors
  1. Saboya Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shuai Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wenhan Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jiaying Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jiaying Liu.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yang, S., Yang, S., Yang, W. et al. Automatic portrait oil painter: joint domain stylization for portrait images. Multimed Tools Appl 77, 16113–16130 (2018). https://doi.org/10.1007/s11042-017-5190-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-017-5190-z

Keywords

Search

Navigation