Advertisement

The Visual Computer

, Volume 35, Issue 3, pp 429–443 | Cite as

Video style transfer by consistent adaptive patch sampling

  • Oriel FrigoEmail author
  • Neus Sabater
  • Julie Delon
  • Pierre Hellier
Original Article
  • 90 Downloads

Abstract

This paper addresses the example-based stylization of videos. Style transfer aims at editing an image so that it matches the style of an example. This topic has been recently investigated by several researchers, both in the industry and in academia. The difficulty lies in how to capture the style of an image and correctly transferring it to a video. In this paper, we build on our previous work “Split and Match” for still pictures, based on adaptive patch synthesis. We address the issue of extending that particular technique to video, ensuring that the solution is spatially and temporally consistent. Results show that our video style transfer is visually plausible, while being very competitive regarding computation time and memory when compared to neural network approaches.

Keywords

Style transfer Texture synthesis Non-photorealistic rendering Video processing 

References

  1. 1.
    Durand, F.: An invitation to discuss computer depiction. In: NPAR, New York, NY, USA, pp. 111–124 (2002).  https://doi.org/10.1145/508530.508550
  2. 2.
    Kyprianidis, J., Collomosse, J., Wang, T., Isenberg, T.: State of the art: a taxonomy of artistic stylization techniques for images and video. IEEE Trans. Vis. Comput. Graph. 19(5), 866–885 (2013)CrossRefGoogle Scholar
  3. 3.
    Gatys, L.A., Ecker, A.S., Bethge, M.: A neural algorithm of artistic style. CoRR. arXiv:1508.06576
  4. 4.
    Li, C., Wand, M.: Precomputed real-time texture synthesis with Markovian generative adversarial networks. In: European Conference on Computer Vision, pp. 702–716. Springer (2016)Google Scholar
  5. 5.
    Dumoulin, V., Shlens, J., Kudlur, M.: A learned representation for artistic style. In: ICLR (2017)Google Scholar
  6. 6.
    Frigo, O., Sabater, N., Delon, J., Hellier, P.: Split and match: example-based adaptive patch sampling for unsupervised style transfer. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June (2016)Google Scholar
  7. 7.
    Efros, A.A., Freeman, W.T.: Image quilting for texture synthesis and transfer. In: SIGGRAPH, New York, NY, USA, pp. 341–346 (2001).  https://doi.org/10.1145/383259.383296
  8. 8.
    Criminisi, A., Perez, P., Toyama, K.: Region filling and object removal by exemplar-based image inpainting. IEEE Trans. Image Process. 13(9), 1200–1212 (2004)CrossRefGoogle Scholar
  9. 9.
    Freeman, W., Jones, T., Pasztor, E.: Example-based super-resolution. IEEE Comput. Graph. Appl. 22(2), 56–65 (2002)CrossRefGoogle Scholar
  10. 10.
    Hertzmann, A., Jacobs, C.E., Oliver, N., Curless, B., Salesin, D.H.: Image analogies. In: SIGGRAPH ’01 Proceedings of the 28th annual conference on Computer graphics and interactive techniques, New York, NY, USA, pp. 327–340 (2001).  https://doi.org/10.1145/383259.383295
  11. 11.
    Reinhard, E., Ashikhmin, M., Gooch, B., Shirley, P.: Color transfer between images. IEEE Comput. Graph. Appl. 21(5), 34–41 (2001).  https://doi.org/10.1109/38.946629 CrossRefGoogle Scholar
  12. 12.
    Efros, A.A., Leung, T.K.: Texture synthesis by non-parametric sampling. In: ICCV, Washington, DC, USA, pp. 1033 (1999). http://dl.acm.org/citation.cfm?id=850924.851569
  13. 13.
    Bénard, P., Cole, F., Kass, M., Mordatch, I., Hegarty, J., Senn, M.S., Fleischer, K., Pesare, D., Breeden, K.: Stylizing animation by example. ACM Trans. Graph. 32(4), 1191–11912 (2013).  https://doi.org/10.1145/2461912.2461929 CrossRefzbMATHGoogle Scholar
  14. 14.
    Barnes, C., Shechtman, E., Goldman, D., Finkelstein, A.: The generalized patchmatch correspondence algorithm. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) Computer Vision ECCV 2010, Series Lecture Notes in Computer Science, vol. 6313, pp. 29–43. Springer, Berlin (2010).  https://doi.org/10.1007/978-3-642-15558-1_3 CrossRefGoogle Scholar
  15. 15.
    Liang, L., Liu, C., Xu, Y.-Q., Guo, B., Shum, H.-Y.: Real-time texture synthesis by patch-based sampling. ACM Trans. Graph. 20(3), 127–150 (2001).  https://doi.org/10.1145/501786.501787 CrossRefGoogle Scholar
  16. 16.
    Shih, Y., Paris, S., Barnes, C., Freeman, W.T., Durand, F.: Style transfer for headshot portraits. ACM Trans. Graph. 33(4), 148 (2014)CrossRefGoogle Scholar
  17. 17.
    Yi, Z., Li, Y., Ji, S., Gong, M.: Artistic stylization of face photos based on a single exemplar. Vis. Comput. 33(11), 1443–1452 (2017).  https://doi.org/10.1007/s00371-016-1290-4 CrossRefGoogle Scholar
  18. 18.
    Freeman, W., Pasztor, E., Carmichael, O.: Learning low-level vision. Int. J. Comput. Vis. 40(1), 25–47 (2000)CrossRefzbMATHGoogle Scholar
  19. 19.
    Wang, X., Tang, X.: Face photo-sketch synthesis and recognition. IEEE Trans. Pattern Anal. Mach. Intell. 31(11), 1955–1967 (2009).  https://doi.org/10.1109/TPAMI.2008.222 CrossRefGoogle Scholar
  20. 20.
    Weiss, Y.: Belief propagation and revision in networks with loops. Technical Report, Cambridge, MA, USA (1997)Google Scholar
  21. 21.
    Elad, M., Milanfar, P.: Style-transfer via texture-synthesis. CoRR (2016). arXiv:1609.03057
  22. 22.
    Ulyanov, D., Vedaldi, A., Lempitsky, V.S.: Instance normalization: The missing ingredient for fast stylization. CoRR (2016). arXiv:1607.08022
  23. 23.
    Puy, G., Kitic, S., Pérez, P.: Unifying local and non-local signal processing with graph CNNS. CoRR (2017). arXiv:1702.07759
  24. 24.
    Joshi, B.J., Stewart, K., Shapiro, D.: Bringing impressionism to life with neural style transfer in come swim. CoRR (2017). arXiv:1701.04928
  25. 25.
    Farbman, Z., Lischinski, D.: Tonal stabilization of video. In: ACM Transactions on Graphics (Proceedings of ACM SIGGRAPH 2011), vol. 30, no. 4, pp. 89:1–89:9 (2011)Google Scholar
  26. 26.
    Frigo, O., Sabater, N., Delon, J., Hellier, P.: Motion driven tonal stabilization. IEEE Trans. Image Process. 25(11), 5455–5468 (2016)MathSciNetCrossRefzbMATHGoogle Scholar
  27. 27.
    Bonneel, N., Tompkin, J., Sunkavalli, K., Sun, D., Paris, S., Pfister, H.: Blind video temporal consistency. In: ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia 2015), vol. 34, no. 6 (2015)Google Scholar
  28. 28.
    Ruder, M., Dosovitskiy, A., Brox, T.: Artistic Style Transfer for Videos. In: Rosenhahn, B., Andres, B. (eds) Pattern Recognition. GCPR 2016. Lecture Notes in Computer Science, vol 9796. Springer, Cham (2016)Google Scholar
  29. 29.
    Fišer, J., Jamriška, O., Lukáč, M., Shechtman, E., Asente, P., Lu, J., Sýkora, D.: Stylit: illumination-guided example-based stylization of 3D renderings. ACM Trans. Graph. 35(4), 92 (2016)Google Scholar
  30. 30.
    Pearl, J.: Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan Kaufmann Publishers Inc., San Francisco (1988)zbMATHGoogle Scholar
  31. 31.
    Frigo, O., Sabater, N., Demoulin, V., Hellier, P.: Optimal transportation for example-guided color transfer. Computer Vision – ACCV 2014, 12th Asian Conference on Computer Vision, Singapore, Singapore, November 1–5, 2014, Revised Selected Papers, Part III (2014)Google Scholar
  32. 32.
    Weinzaepfel, P., Revaud, J., Harchaoui, Z., Schmid, C.: DeepFlow: large displacement optical flow with deep matching. In: IEEE International Conference on Computer Vision (ICCV), Sydney, Australia, December (2013). http://hal.inria.fr/hal-00873592
  33. 33.
    Sundaram, N., Brox, T., Keutzer, K.: Dense point trajectories by GPU-accelerated large displacement optical flow. In: European Conference on Computer Vision (ECCV), Series Lecture Notes in Computer Science. Springer, September (2010). http://lmb.informatik.uni-freiburg.de//Publications/2010/Bro10e
  34. 34.
    Butler, D.J., Wulff, J., Stanley, G.B., Black, M.J.: A naturalistic open source movie for optical flow evaluation. In: Fitzgibbon, A., et al. (eds.) European Conference on Computer Vision (ECCV), Series Part IV, LNCS 7577, pp. 611–625. Springer, Berlin (2012)Google Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Technicolor R&ICesson-SévignéFrance
  2. 2.Université Paris DescartesParisFrance

Personalised recommendations