Advertisement

Multimedia Tools and Applications

, Volume 78, Issue 14, pp 19565–19586 | Cite as

Photographic painting style transfer using convolutional neural networks

  • Asad Khan
  • Muhammad Ahmad
  • Nuzhat Naqvi
  • Faisal Yousafzai
  • Jing XiaoEmail author
Article
  • 391 Downloads

Abstract

We propose a novel automatic photographic painting style technique with a single example image by using Convolutional Neural Networks (CNN). The photographic painting style is a challenging problem in the research community. Even though, researchers have been trying to obtain good results on painting style, but not much has been done on photographic stylization. Portrait painting techniques are mainly designed for the graphite style and/or are based on image analogies; an example painting as well as its original unpainted version are required. This preceding issue is a motivation of our proposed methods. As a result, our method extends the limits of their domain of applicability. We present a novel multi-convolutional-learning technique that is developed for both images (NPR/PR) labeling, style transmission and elevating a particular unified CNN model per weight sharing. A new painting technique is generated that follows the example style in the example image and maintains the integrity of facial structures. We believe this novel interpretation connects these two important research fields and could enlighten future researches. Moreover, our proposed technique is not restricted to headshot images or specific styles as our method can also change the photographic painting style in the wild.

Keywords

Photographics Painting Style (PPS) Non-Photorealistic Rendering (NPR) Photorealistic Rendering (PR) Digital painting Example-based painting 

Notes

Acknowledgements

This work was supported by the National Natural Science Foundation of China (Grant No. 61202296, 61872153) and the National Science Foundation of Guangdong province No. 2018A030313318.

References

  1. 1.
  2. 2.
    Printerest (2017) https://www.pinterest.com/
  3. 3.
    Ashikhmin M (2001) Synthesizing natural textures. In: Proceedings of the 2001 symposium on interactive 3D graphics. ACM, pp 217–226Google Scholar
  4. 4.
    Ashikhmin N (2003) Fast texture transfer. IEEE Comput Graph Appl 23(4):38–43CrossRefGoogle Scholar
  5. 5.
    Chen H, Liang L, Xu Y-Q, Shum H-Y, Zheng N-N (2003) Example-based automatic portraiture. Chinese journal of computers-chinese edition- 26 2:147–152Google Scholar
  6. 6.
    Chen H, Liu Z, Rose C, Xu Y, Shum H-Y, Salesin D (2004) Example-based composite sketching of human portraits. In: Proceedings of the 3rd international symposium on Non-photorealistic animation and rendering. ACM, pp 95–153Google Scholar
  7. 7.
    Chen H, Xu Y-Q, Shum H-Y, Zhu S-C, Zheng N-N (2001) Example-based facial sketch generation with non-parametric sampling. In: Proceedings 8th IEEE international conference on computer vision, ICCV 2001, vol 2. IEEE, pp 433–438Google Scholar
  8. 8.
    Chen H, Zheng N-N, Liang L, Li Y, Xu Y-Q, Shum H-Y (2002) Pictoon: a personalized image-based cartoon system. In: Proceedings of the tenth ACM international conference on multimedia. ACM, pp 171–178Google Scholar
  9. 9.
    Collomosse JP, Hall PM (2005) Genetic paint: A search for salient paintings. In: Workshops on applications of evolutionary computation. Springer, pp 437–447Google Scholar
  10. 10.
    Efros AA, Freeman WT (2001) Image quilting for texture synthesis and transfer. In: Proceedings of the 28th annual conference on Computer graphics and interactive techniques. ACM, pp 341–346Google Scholar
  11. 11.
    Efros AA, Leung TK (1999) Texture synthesis by non-parametric sampling. In: The proceedings of the 7th IEEE international conference on computer vision, 1999, vol 2. IEEE, pp 1033–1038Google Scholar
  12. 12.
    Farabet C, Couprie C, Najman L, LeCun Y (2013) Learning hierarchical features for scene labeling. IEEE Trans Pattern Anal Mach Intell 35(8):1915–1929CrossRefGoogle Scholar
  13. 13.
    Gao L, Guo Z, Zhang H, Xu X, Shen HT (2017) Video captioning with attention-based lstm and semantic consistency. IEEE Trans Multimed 19(9):2045–2055CrossRefGoogle Scholar
  14. 14.
    Gatys L, Ecker AS, Bethge M (2015) Texture synthesis using convolutional neural networks. In: Advances in neural information processing systems, pp 262–270Google Scholar
  15. 15.
    Gatys LA, Ecker AS, Bethge M (2016) Image style transfer using convolutional neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2414– 2423Google Scholar
  16. 16.
    Gooch B, Reinhard E, Gooch A (2004) Human facial illustrations creation and psychophysical evaluation. ACM Trans Graph 23(1):27–44CrossRefGoogle Scholar
  17. 17.
    Gu S, Chen C, Liao J, Yuan L (2018) Arbitrary style transfer with deep feature reshuffle. CoRR: 1805.04103
  18. 18.
    Hertzmann A (1998) Painterly rendering with curved brush strokes of multiple sizes. In: Proceedings of the 25th annual conference on Computer graphics and interactive techniques. ACM, pp 453– 460Google Scholar
  19. 19.
    Hertzmann A, Jacobs CE, Oliver N, Curless B, Salesin DH (2001) Image analogies. In: Proceedings of the 28th annual conference on Computer graphics and interactive techniques. ACM, pp 327–340Google Scholar
  20. 20.
    Huang X, Belongie S (2017) Arbitrary style transfer in real-time with adaptive instance normalization. In: IEEE international conference on computer vision (ICCV)Google Scholar
  21. 21.
    Johnson J, Alahi A, Fei-Fei L (2016) Perceptual losses for real-time style transfer and super-resolution. In: European conference on computer vision. Springer, pp 694–711Google Scholar
  22. 22.
    Kim SY, Maciejewski R, Isenberg T, Andrews WM, Chen W, Sousa MC, Ebert DS (2009) Stippling by example. In: Proceedings of the 7th international symposium on non-photorealistic animation and rendering. ACM, pp 41–50Google Scholar
  23. 23.
    Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Pereira F, Burges CJC, Bottou L, Weinberger KQ (eds) Advances in neural information processing systems 25. Curran Associates, Inc., pp 1097–1105Google Scholar
  24. 24.
    Kwatra V, Essa I, Bobick A, Kwatra N (2005) Texture optimization for example-based synthesis. ACM Trans Graph 24(3):795–802CrossRefGoogle Scholar
  25. 25.
    Kyprianidis JE, Collomosse J, Wang T, Isenberg T (2013) State of the art: a taxonomy of artistic stylization techniques for images and video. IEEE Trans Vis Comput Graph 19(5):866–885CrossRefGoogle Scholar
  26. 26.
    Ledig C, Theis L, Huszár F, Caballero J, Cunningham A, Acosta A, Aitken A, Tejani A, Totz J, Wang Z et al (2016) Photo-realistic single image super-resolution using a generative adversarial network. arXiv:1609.04802
  27. 27.
    Lee H, Seo S, Ryoo S, Yoon K (2010) Directional texture transfer. In: Proceedings of the 8th international symposium on non-photorealistic animation and rendering. ACM, pp 43–48Google Scholar
  28. 28.
    Li C, Wand M (2016) Combining markov random fields and convolutional neural networks for image synthesis. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR), pp 2479– 2486Google Scholar
  29. 29.
    Li Y, Fang C, Yang J, Wang Z, Lu X, Yang M-H (2017) Universal style transfer via feature transforms. In: Advances in neural information processing systemsGoogle Scholar
  30. 30.
    Liao J, Yao Y, Yuan L, Hua G, Kang SB (2017) Visual attribute transfer through deep image analogy. ACM Trans Graph 36(4):120:1–120:15CrossRefGoogle Scholar
  31. 31.
    Lu M, Zhao H, Yao A, Xu F, Chen Y, Zhang L (2017) Decoder network over lightweight reconstructed feature for fast semantic style transfer. In: 2017 IEEE international conference on computer vision (ICCV), pp 2488–2496Google Scholar
  32. 32.
    McKone E, Kanwisher N, Duchaine BC (2007) Can generic expertise explain special processing for faces? Trends Cogn Sci 11(1):8–15CrossRefGoogle Scholar
  33. 33.
    Meng M, Zhao M, Zhu S-C (2010) Artistic paper-cut of human portraits. In: Proceedings of the 18th ACM international conference on multimedia. ACM, pp 931–934Google Scholar
  34. 34.
    Pinheiro PHO, Collobert R (2013) Recurrent convolutional neural networks for scene parsing. CoRR 1306.2795
  35. 35.
    Ruder M, Dosovitskiy A, Brox T (2016) Artistic style transfer for videos. In: German conference on pattern recognition. Springer, pp 26–36Google Scholar
  36. 36.
    Selim A, Elgharib M, Doyle L (2016) Painting style transfer for head portraits using convolutional neural networks. ACM Trans Graph 35(4):129CrossRefGoogle Scholar
  37. 37.
    Shih Y, Paris S, Barnes C, Freeman WT, Durand F (2014) Style transfer for headshot portraits. ACM Trans Graph 33(4):148:1–148:14CrossRefGoogle Scholar
  38. 38.
    Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556
  39. 39.
    Sinha P, Balas B, Ostrovsky Y, Russell R (2006) Face recognition by humans Nineteen results all computer vision researchers should know about. Proc IEEE 94(11):1948–1962CrossRefGoogle Scholar
  40. 40.
    Song J, Gao L, Nie F, Shen HT, Yan Y, Sebe N (2016) Optimized graph learning using partial tags and multiple features for image and video annotation. IEEE Trans Image Process 25(11):4999– 5011MathSciNetCrossRefzbMATHGoogle Scholar
  41. 41.
    Song J, Guo Y, Gao L, Li X, Hanjalic A, Shen HT (2018) From deterministic to generative: multi-modal stochastic RNNs for video captioning. IEEE Trans Neural Netw Learn Syst.  https://doi.org/10.1109/TNNLS.2018.2851077
  42. 42.
    Song J, Zhang H, Li X, Gao L, Wang M, Hong R (2018) Self-supervised video hashing with hierarchical binary auto-encoder. IEEE Trans Image Process 27 (7):3210–3221MathSciNetCrossRefzbMATHGoogle Scholar
  43. 43.
    Tompson J, Jain A, LeCun Y, Bregler C (2014) Joint training of a convolutional network and a graphical model for human pose estimation. In: Proceedings of the 27th international conference on neural information processing systems, NIPS’14. MIT Press, Cambridge, pp 1799–1807Google Scholar
  44. 44.
    Ulyanov D, Lebedev V, Vedaldi A, Lempitsky V (2016) Texture networks Feed-forward synthesis of textures and stylized images. In: Interational conference on machine learning (ICML)Google Scholar
  45. 45.
    Wang B, Wang W, Yang H, Sun J (2004) Efficient example-based painting and synthesis of 2d directional texture. IEEE Trans Vis Comput Graph 10(3):266–277CrossRefGoogle Scholar
  46. 46.
    Wang T, Collomosse JP, Hunter A, Greig D (2013) Learnable stroke models for example-based portrait painting. In: British machine vision conferenceGoogle Scholar
  47. 47.
    Wang X, Gao L, Song J, Shen H (2017) Beyond frame-level cnn Saliency-aware 3-d cnn with lstm for video action recognition. IEEE Signal Process Lett 24(4):510–514CrossRefGoogle Scholar
  48. 48.
    Wang X, Gao L, Wang P, Sun X, Liu X Two-stream 3-d convnet fusion for action recognition in videos with arbitrary size and length. 1–1Google Scholar
  49. 49.
    Wang X, Tang X (2009) Face photo-sketch synthesis and recognition. IEEE Trans Pattern Anal Mach Intell 31(11):1955–1967CrossRefGoogle Scholar
  50. 50.
    Zeng K, Zhao M, Xiong C, Zhu S-C (2009) From image parsing to painterly rendering. ACM Trans Graph 29(1):2CrossRefGoogle Scholar
  51. 51.
    Zhao M, Zhu S-C (2011) Portrait painting using active templates. In: Proceedings of the ACM SIGGRAPH/Eurographics symposium on non-photorealistic animation and rendering. ACM, pp 117– 124Google Scholar
  52. 52.
    Zhu C, Byrd RH, Lu P, Nocedal J (1997) Algorithm 778: L-bfgs-b: Fortran subroutines for large-scale bound-constrained optimization. ACM Trans Math Softw 23 (4):550–560MathSciNetCrossRefzbMATHGoogle Scholar
  53. 53.
    Zhu X, Li X, Zhang S (2016) Block-row sparse multiview multilabel learning for image classification. IEEE Trans Cybern 46:450–461CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.School of Computer ScienceSouth China Normal UniversityGuangzhouPeople’s Republic of China
  2. 2.Institute of Robotics, Computer Science DepartmentInnopolis UniversityInnopolisRussia
  3. 3.School of Electronic Engineering and Information ScienceUniversity of Science and Technology of ChinaHefeiPeople’s Republic of China
  4. 4.Military College of EngineeringNational University of Sciences and Technology (NUST)IslamabadPakistan

Personalised recommendations