Abstract
The objective of this paper is to develop an effective photographic transfer method while preserving the semantic correspondence between the style and content images. A semantic correspondence guided deep photo style transfer algorithm is developed, which is to ensure that the semantic structure of the content image has not been changed while the color of the style images is being migrated. The semantic correspondence is constructed in large scale regions based on image segmentation and also in local scale patches using deep image analogy. Based on the semantic correspondence, a matting optimization is utilized to optimize the style transfer result to ensure the semantic accuracy and transfer faithfulness. The proposed style transfer method is further extended to automatically retrieve the style images from a database to make style transfer more-friendly. The experimental results show that our method could successfully conduct the style transfer while preserving semantic correspondence between diversity of scenes. A user study also shows that our method outperforms state-of-the-art photographic style transfer methods.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Barnes, C., Shechtman, E., Finkelstein, A., Goldman, D.B.: Patchmatch: a randomized correspondence algorithm for structural image editing. ACM Trans. Graph. 28, 3 (2009). (Proc. of SIGGRAPH)
Champandard, A.J.: Semantic style transfer and turning two-bit doodles into fine artworks. arXiv preprint arXiv:1603.01768 (2016)
Gatys, L.A., Ecker, A.S., Bethge, M.: A neural algorithm of artistic style. In: Computer Science, Computer Vision and Pattern Recognition (2015)
Hertzmann, A., Jacobs, C.E., Oliver, N., Curless, B., Salesin, D.H.: Image analogies. In: Conference on Computer Graphics and Interactive Techniques, pp. 327–340 (2001)
Hwang, S.J., Kapoor, A., Kang, S.B.: Context-based automatic local image enhancement. In: European Conference on Computer Vision, pp. 569–582 (2012)
Kang, S.B.: Visual attribute transfer through deep image analogy. ACM Trans. Graph. 36(4), 120 (2017)
Levin, A., Lischinski, D., Weiss, Y.: A closed-form solution to natural image matting. IEEE Trans. Pattern Anal. Mach. Intell. 30(2), 228–242 (2008)
Li, C., Wand, M.: Combining Markov random fields and convolutional neural networks for image synthesis. In: Computer Vision and Pattern Recognition, pp. 2479–2486 (2016)
Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Computer Vision and Pattern Recognition, pp. 3431–3440 (2015)
Luan, F., Paris, S., Shechtman, E., Bala, K.: Deep photo style transfer. In: Computer Vision and Pattern Recognition, pp. 6997–7005 (2017)
Pitie, F., Kokaram, A.C., Dahyot, R.: N-dimensional probability density function transfer and its application to color transfer. In: Tenth IEEE International Conference on Computer Vision, pp. 1434–1439 (2005)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
Acknowledgement
The authors wish to acknowledge the financial support from: (i) Chinese Natural Science Foundation under the Grant No. 61602313, 61620106008; (ii) Shenzhen Commission of Scientific Research and Innovations under the Grant No. JCYJ20170302153632883, JCYJ20160422151736824; (iii) Tencent “Rhinoceros Birds” - Scientific Research Foundation for Young Teachers of Shenzhen University; (iv) Startup Foundation for Advanced Talents, Shenzhen; (v) The Natural Science Foundation of Guangdong Province No. 2016A030310053, 2017A030310521.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Xiao, Z., Zhang, X., Zhang, X. (2018). Semantic Correspondence Guided Deep Photo Style Transfer. In: Hong, R., Cheng, WH., Yamasaki, T., Wang, M., Ngo, CW. (eds) Advances in Multimedia Information Processing – PCM 2018. PCM 2018. Lecture Notes in Computer Science(), vol 11164. Springer, Cham. https://doi.org/10.1007/978-3-030-00776-8_8
Download citation
DOI: https://doi.org/10.1007/978-3-030-00776-8_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-00775-1
Online ISBN: 978-3-030-00776-8
eBook Packages: Computer ScienceComputer Science (R0)