Advertisement

Springer Nature is making SARS-CoV-2 and COVID-19 research free. View research | View latest news | Sign up for updates

SmartPaint: a co-creative drawing system based on generative adversarial networks

Abstract

Artificial intelligence (AI) has played a significant role in imitating and producing large-scale designs such as e-commerce banners. However, it is less successful at creative and collaborative design outputs. Most humans express their ideas as rough sketches, and lack the professional skills to complete pleasing paintings. Existing AI approaches have failed to convert varied user sketches into artistically beautiful paintings while preserving their semantic concepts. To bridge this gap, we have developed SmartPaint, a co-creative drawing system based on generative adversarial networks (GANs), enabling a machine and a human being to collaborate in cartoon landscape painting. SmartPaint trains a GAN using triples of cartoon images, their corresponding semantic label maps, and edge detection maps. The machine can then simultaneously understand the cartoon style and semantics, along with the spatial relationships among the objects in the landscape images. The trained system receives a sketch as a semantic label map input, and automatically synthesizes its edge map for stable handling of varied sketches. It then outputs a creative and fine painting with the appropriate style corresponding to the human’s sketch. Experiments confirmed that the proposed SmartPaint system successfully generates high-quality cartoon paintings.

This is a preview of subscription content, log in to check access.

References

  1. Atarsaikhan G, Iwana BK, Narusawa A, et al., 2017. Neural font style transfer. Proc 14th IAPR Int Conf on Document Analysis and Recognition, p.51–56. https://doi.org/10.1109/ICDAR.2017.328

  2. Belongie S, Malik J, Puzicha J, 2001. Shape context: a new descriptor for shape matching and object recognition. Proc 13th Int Conf on Neural Information Processing Systems, p.798–804.

  3. Benedetti L, Winnemöller H, Corsini M, et al., 2014. Painting with Bob: assisted creativity for novices. Proc 27th Annual ACM Symp on User Interface Software and Technology, p.419–428. https://doi.org/10.1145/2642918.2647415

  4. Bowman SR, Vilnis L, Vinyals O, et al., 2016. Generating sentences from a continuous space. Proc 20th SIGNLL Conf on Computational Natural Language Learning, p.10–21. https://doi.org/10.18653/v1/K16-1002

  5. Canny J, 1987. A computational approach to edge detection. In: Fischler MA, Firschein O (Eds.), Readings in Computer Vision: Issues, Problem, Principles, and Paradigms. Elsevier, Amsterdam, p.184–203. https://doi.org/10.1016/B978-0-08-051581-6.50024-6

  6. Champandard AJ, 2016. Semantic style transfer and turning two-bit doodles into fine artworks. https://doi.org/1603.01768

  7. Chen C, Lin JC, Liao MH, et al., 2016. Learning to detect salient curves of cartoon images based on composition rules. Proc 11th Int Conf on Computer Science & Education, p.808–813. https://doi.org/10.1109/ICCSE.2016.7581686

  8. Chen T, Cheng MM, Tan P, et al., 2009. Sketch2Photo: Internet image montage. ACM Trans Graph, 28(5), Article 124. https://doi.org/10.1145/1618452.1618470

  9. Chen Y, Lai YK, Liu YJ, 2018. CartoonGAN: generative adversarial networks for photo cartoonization. IEEE/CVF Conf on Computer Vision and Pattern Recognition, p.9465–9474. https://doi.org/10.1109/CVPR.2018.00986

  10. Chu NSH, Tai CL, 2004. Real-time painting with an expressive virtual Chinese brush. IEEE Comput Graph Appl, 24(5):76–85. https://doi.org/10.1109/MCG.2004.37

  11. Ci YZ, Ma XZ, Wang ZH, et al., 2018. User-guided deep anime line art colorization with conditional adversarial networks. Proc 26th ACM Int Conf on Multimedia, p.1536–1544. https://doi.org/10.1145/3240508.3240661

  12. Cummmings D, Vides F, Hammond T, 2012. I don’t believe my eyes! Geometric sketch recognition for a computer art tutorial. Proc Int Symp on Sketch-Based Interfaces and Modeling, p.97–106. https://doi.org/10.2312/SBM/SBM12/097-106

  13. Davis NM, 2013. Human-computer co-creativity: blending human and computational creativity. Proc 9th Artificial Intelligence and Interactive Digital Entertainment Conf, p.9–12.

  14. Davis NM, Hsiao CP, Singh KY, et al., 2015. Drawing apprentice: an enactive co-creative agent for artistic collaboration. Proc ACM SIGCHI Conf on Creativity and Cognition, p.185–186. https://doi.org/10.1145/2757226.2764555

  15. Davis NM, Hsiao CP, Singh KY, et al., 2016a. Co-creative drawing agent with object recognition. Proc 12th Artificial Intelligence and Interactive Digital Entertainment Conf, p.9–15.

  16. Davis NM, Hsiao CP, Yashraj Singh K, et al., 2016b. Empirically studying participatory sense-making in abstract drawing with a co-creative cognitive agent. Proc 21st Int Conf on Intelligent User Interfaces, p.196–207. https://doi.org/10.1145/2856767.2856795

  17. Dekel T, Gan C, Krishnan D, et al., 2018. Sparse, smart contours to represent and edit images. Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition, p.3511–3520. https://doi.org/10.1109/CVPR.2018.00370

  18. Dixon D, Prasad M, Hammond T, 2010. iCanDraw: using sketch recognition and corrective feedback to assist a user in drawing human faces. Proc SIGCHI Conf on Human Factors in Computing Systems, p.897–906. https://doi.org/10.1145/1753326.1753459

  19. Eitz M, Richter R, Hildebrand K, et al., 2011. Photosketcher: interactive sketch-based image synthesis. IEEE Comput Graph Appl, 31(6):56–66. https://doi.org/10.1109/MCG.2011.67

  20. Gatys LA, Ecker AS, Bethge M, 2016. Image style transfer using convolutional neural networks. Proc IEEE Conf on Computer Vision and Pattern Recognition, p.2414–2423. https://doi.org/10.1109/CVPR.2016.265

  21. Goodfellow I, Pouget-Abadie J, Mirza M, et al., 2014. Generative adversarial nets. Proc 27th Int Conf on Neural Information Processing Systems, p.2672–2680.

  22. Güçlütürk Y, Güçlü U, van Lier R, et al., 2016. Convolutional sketch inversion. European Conf on Computer Vision, p.810–824. https://doi.org/10.1007/978-3-319-46604-0_56

  23. Ha D, Eck D, 2017. A neural representation of sketch drawings. https://doi.org/1704.03477

  24. Huang HZ, Wang H, Luo WH, et al., 2017. Real-time neural style transfer for videos. Proc IEEE Conf on Computer Vision and Pattern Recognition, p.783–791. https://doi.org/10.1109/CVPR.2017.745

  25. Isola P, Zhu JY, Zhou TH, et al., 2017. Image-to-image translation with conditional adversarial networks. Proc IEEE Conf on Computer Vision and Pattern Recognition, p.1125–1134. https://doi.org/10.1109/CVPR.2017.632

  26. Karimi P, Davis N, Grace K, et al., 2018. Deep learning for identifying potential conceptual shifts for co-creative drawing. https://doi.org/1801.00723

  27. Lee YJ, Zitnick CL, Cohen MF, 2011. ShadowDraw: realtime user guidance for freehand drawing. ACM Trans Graph, 30(4), Article 27. https://doi.org/10.1145/2010324.1964922

  28. Li MJ, Huang HZ, Ma L, et al., 2018. Unsupervised image-to-image translation with stacked cycle-consistent adversarial networks. Proc 15th European Conf on Computer Vision, p.186–201. https://doi.org/10.1007/978-3-030-01240-3_12

  29. Liu YF, Qin ZC, Wan T, et al., 2018. Auto-painter: cartoon image generation from sketch by using conditional Wasserstein generative adversarial networks. Neurocomputing, 311:78–87. https://doi.org/10.1016/j.neucom.2018.05.045

  30. Luan FJ, Paris S, Shechtman E, et al., 2017. Deep photo style transfer. Proc IEEE Conf on Computer Vision and Pattern Recognition, p.4990–4998. https://doi.org/10.1109/CVPR.2017.740

  31. Mirza M, Osindero S, 2014. Conditional generative adversarial nets. https://doi.org/1411.1784

  32. Ning X, Laga H, Saito S, et al., 2011. Contour-driven Sumi-e rendering of real photos. Comput Graph, 35(1):122–134. https://doi.org/10.1016/j.cag.2010.11.017

  33. Oh C, Song J, Choi J, et al., 2018. I lead, you help but only with enough details: understanding user experience of co-creation with artificial intelligence. Proc CHI Conf on Human Factors in Computing Systems, Article 649. https://doi.org/10.1145/3173574.3174223

  34. Portenier T, Hu QY, Szabó A, et al., 2018. Faceshop: deep sketch-based face image editing. ACM Trans Graph, 37(4), Article 99. https://doi.org/10.1145/3197517.3201393

  35. Roberts A, Engel J, Eck D, 2017. Hierarchical variational autoencoders for music. Workshop on Machine Learning for Creativity and Design, NIPS.

  36. Sangkloy P, Lu JW, Fang C, et al., 2017. Scribbler: controlling deep image synthesis with sketch and color. Proc IEEE Conf on Computer Vision and Pattern Recognition, p.5400–5409. https://doi.org/10.1109/cvpr.2017.723

  37. Selim A, Elgharib M, Doyle L, 2016. Painting style transfer for head portraits using convolutional neural networks. ACM Trans Graph, 35(4), Article 129. https://doi.org/10.1145/2897824.2925968

  38. Simo-Serra E, Iizuka S, Sasaki K, et al., 2016. Learning to simplify: fully convolutional networks for rough sketch cleanup. ACM Trans Graph, 35(4), Article 121. https://doi.org/10.1145/2897824.2925972

  39. Wang TC, Liu MY, Zhu JY, et al., 2018. High-resolution image synthesis and semantic manipulation with conditional GANs. Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition, p.8798–8807. https://doi.org/10.1109/CVPR.2018.00917

  40. Xian WQ, Sangkloy P, Agrawal V, et al., 2018. TextureGAN: controlling deep image synthesis with texture patches. Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition, p.8456–8465. https://doi.org/10.1109/cvpr.2018.00882

  41. Zhang YK, Hu KK, Ren PR, et al., 2017. Layout style modeling for automating banner design. Proc Thematic Workshops of ACM Multimedia, p.451–459. https://doi.org/10.1145/3126686.3126718

  42. Zhao NX, Cao Y, Lau RWH, 2018. What characterizes personalities of graphic designs? ACM Trans Graph, 37(4), Article 116. https://doi.org/10.1145/3197517.3201355

  43. Zhu JY, Krähenbühl P, Shechtman E, et al., 2016. Generative visual manipulation on the natural image manifold. Proc 14th European Conf on Computer Vision, p.597–613. https://doi.org/10.1007/978-3-319-46454-1_36

  44. Zhu JY, Park T, Isola P, et al., 2017. Unpaired image-to-image translation using cycle-consistent adversarial networks. Proc IEEE Int Conf on Computer Vision, p.2223–2232. https://doi.org/10.1109/ICCV.2017.244

Download references

Author information

Correspondence to Wei Xiang.

Additional information

Compliance with ethics guidelines

Lingyun SUN, Pei CHEN, Wei XIANG, Peng CHEN, Wei-yue GAO, and Ke-jun ZHANG declare that they have no conflict of interest.

Project supported by the National Science and Technology Innovation 2030 Major Project of the Ministry of Science and Technology of China (No. 2018AAA0100703), the National Natural Science Foundation of China (No. 61672451), the Provincial Key Research and Development Plan of Zhejiang Province, China (No. 2019C03137), the China Postdoctoral Science Foundation (No. 2018M630658), and the Ng Teng Fong Charitable Foundation in the form of ZJU-SUTD IDEA Grant

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Sun, L., Chen, P., Xiang, W. et al. SmartPaint: a co-creative drawing system based on generative adversarial networks. Front Inform Technol Electron Eng 20, 1644–1656 (2019). https://doi.org/10.1631/FITEE.1900386

Download citation

Key words

  • Co-creative drawing
  • Deep learning
  • Image generation

CLC number

  • TP391