Anime Sketch Coloring with Swish-Gated Residual U-Net

  • Gang LiuEmail author
  • Xin Chen
  • Yanzhong Hu
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 986)


Anime sketch coloring is to fill the color into the anime sketches to obtain the colorful anime images and it is a new research direction in deep learning technology. Currently, generative adversarial networks (GANs) have been used for anime sketch coloring and achieved some results. However, the colorful images generated by the anime sketch coloring methods based on GANs generally have poor coloring effects. In this paper, an efficient anime sketch coloring method based on swish-gated residual U-net (SGRU) is proposed to solve the above problems. In SGRU, the proposed swish layer and swish-gated residual blocks (SGRBs) effectively filter the information transmitted by each level and speed up the convergence of the network. The perceptual loss and the per-pixel loss are used to constitute the final loss of SGRU. The final loss function reflects the coloring results more realistically and can control the effect of coloring more effectively. SGRU can automatically color the sketch without providing any coloring hints in advance and can be trained end-to-end with the sketch and the corresponding color image. Experimental results show that our method performs better than other state-of-the-art coloring methods, and can achieve the colorful images with higher visual quality.


Anime sketch coloring U-net Swish layer Swish-gated residual blocks 



The work described in this paper was support by National Natural Science Foundation of China Foundation No. 61300127. Any conclusions or recommendations stated here are those of the authors and do not necessarily reflect official positions of NSFC.


  1. 1.
    Anonymous, The Danbooru Community, Branwen, G., Gokaslan, A.: Danbooru2017: a large-scale crowdsourced and tagged anime illustration dataset.
  2. 2.
    Ba, J.L., Kiros, J.R., Hinton, G.E.: Layer normalization. CoRR abs/1607.06450 (2016).
  3. 3.
    Chen, Q., Koltun, V.: Photographic image synthesis with cascaded refinement networks. In: Proceedings of International Conference on Computer Vision 2017 (ICCV 2017), Venice, Italy, pp. 1511–1520, October 2017Google Scholar
  4. 4.
    Dong, H., Yang, G., Liu, F., Mo, Y., Guo, Y.: Automatic brain tumor detection and segmentation using U-Net based fully convolutional networks. In: Valdés Hernández, M., González-Castro, V. (eds.) MIUA 2017. CCIS, vol. 723, pp. 506–517. Springer, Cham (2017). Scholar
  5. 5.
    He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: surpassing human-level performance on imagenet classification. In: Proceedings of 2015 IEEE International Conference on Computer Vision (ICCV 2015), Santiago, Chile, pp. 1026–1034 (2015)Google Scholar
  6. 6.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of 29th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, United states, pp. 770–778 (2016)Google Scholar
  7. 7.
    Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: Proceedings of 30th IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017), Honolulu, HI, United States, pp. 5967–5976, July 2017Google Scholar
  8. 8.
    Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 694–711. Springer, Cham (2016). Scholar
  9. 9.
    Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: Proceedings of the 3rd International Conference for Learning Representations (ICLR 2015), San Diego, CA, United States, pp. 1–15, May 2015Google Scholar
  10. 10.
    Lin, B.S., Michael, K., Kalra, S., Tizhoosh, H.R.: Skin lesion segmentation: U-nets versus clustering. In: Proceedings of 2017 IEEE Symposium Series on Computational Intelligence (SSCI 2017), Honolulu, HI, United States, pp. 1–7, November 2017Google Scholar
  11. 11.
    Liu, Y., Qin, Z., Wan, T., Luo, Z.: Auto-painter: cartoon image generation from sketch by using conditional Wasserstein generative adversarial networks. Neurocomputing 311, 78–87 (2018)CrossRefGoogle Scholar
  12. 12.
    Ramachandran, P., Zoph, B., Le, Q.V.: Searching for activation functions. CoRR abs/1710.05941 (2017).
  13. 13.
    Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). Scholar
  14. 14.
    Schmidhuber, J.: Deep learning in neural networks: an overview. Neural Netw. 61, 85–117 (2015)CrossRefGoogle Scholar
  15. 15.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. CoRR abs/1409.1556 (2014).
  16. 16.
    Wang, K., Gou, C., Duan, Y., Lin, Y., Zheng, X., Wang, F.Y.: Generative adversarial networks: introduction and outlook. IEEE/CAA J. Autom. Sinica 4(4), 588–598 (2017)MathSciNetCrossRefGoogle Scholar
  17. 17.
    Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)CrossRefGoogle Scholar
  18. 18.
  19. 19.
    Zhang, L., Ji, Y., Lin, X.: Style transfer for anime sketches with enhanced residual U-net and auxiliary classifier GAN. In: Proceedings of Asian Conference on Pattern Recognition (ACPR 2017), Nanjing, China, November 2017Google Scholar
  20. 20.
    Zhang, Z., Liu, Q., Wang, Y.: Road extraction by deep residual U-net. IEEE Geosci. Remote Sens. Lett. 15(5), 749–753 (2018)CrossRefGoogle Scholar
  21. 21.
    Zhao, H., Sun, N.: Improved U-net model for nerve segmentation. In: Zhao, Y., Kong, X., Taubman, D. (eds.) ICIG 2017. LNCS, vol. 10667, pp. 496–504. Springer, Cham (2017). Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2019

Authors and Affiliations

  1. 1.School of Computer ScienceHubei University of TechnologyWuhanChina

Personalised recommendations