Advertisement

Automatic Video Colorization Using 3D Conditional Generative Adversarial Networks

  • Panagiotis Kouzouglidis
  • Giorgos SfikasEmail author
  • Christophoros Nikou
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11844)

Abstract

In this work, we present a method for automatic colorization of grayscale videos. The core of the method is a Generative Adversarial Network that is trained and tested on sequences of frames in a sliding window manner. Network convolutional and deconvolutional layers are three-dimensional, with frame height, width and time as the dimensions taken into account. Multiple chrominance estimates per frame are aggregated and combined with available luminance information to recreate a colored sequence. Colorization trials are run successfully on a dataset of old black-and-white films. The usefulness of our method is also validated with numerical results, computed with a newly proposed metric that measures colorization consistency over a frame sequence.

Keywords

Video colorization Generative Adversarial Networks Three-dimensional convolution Black-and-white films 

Notes

Acknowledgements

We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan XP GPU used for this research.

References

  1. 1.
    Ben-Zrihem, N., Zelnik-Manor, L.: Approximate nearest neighbor fields in video. In: IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5233–5242 (2015)Google Scholar
  2. 2.
    Chintala, S., Denton, E., Arjovsky, M., Mathieu, M.: How to train a GAN? Tips and tricks to make GANs work (2016). http://github.com/soumith/ganhacks. Accessed 25 January 2018
  3. 3.
    Daskalakis, C., Ilyas, A., Syrgkanis, V., Zeng, H.: Training GANs with optimism. CoRR abs/1711.00141 (2017). http://arxiv.org/abs/1711.00141
  4. 4.
    Goodfellow, I., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems (NIPS), pp. 2672–2680 (2014)Google Scholar
  5. 5.
    Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. arXiv preprint arXiv:1611.07004 (2016)
  6. 6.
    Juliani, A.: Pix2Pix-Film (2017). http://github.com/awjuliani/Pix2Pix-Film. Accessed 2 January 2018
  7. 7.
    Levin, A., Lischinski, D., Weiss, Y.: Colorization using optimization. ACM Trans. Graph. (TOG) 23, 689–694 (2004)CrossRefGoogle Scholar
  8. 8.
    Meyer, S., Cornillère, V., Djelouah, A., Schroers, C., Gross, M.: Deep video color propagation. arXiv preprint arXiv:1808.03232 (2018)
  9. 9.
    Otani, M., Hioki, H.: Video colorization based on optical flow and edge-oriented color propagation. In: Computational Imaging XII. vol. 9020, p. 902002. International Society for Optics and Photonics (2014)Google Scholar
  10. 10.
    Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015).  https://doi.org/10.1007/978-3-319-24574-4_28CrossRefGoogle Scholar
  11. 11.
    Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., Chen, X.: Improved techniques for training GANs. In: Advances in neural information processing systems (NIPS), pp. 2234–2242 (2016)Google Scholar
  12. 12.
    Sheng, B., Sun, H., Magnor, M., Li, P.: Video colorization using parallel optimization in feature space. IEEE Trans. Circuits Syst. Video Technol. 24(3), 407–417 (2014)CrossRefGoogle Scholar
  13. 13.
    Veeravasarapu, V.R., Sivaswamy, J.: Fast and fully automated video colorization. In: 2012 International Conference on Signal Processing and Communications (SPCOM), pp. 1–5. IEEE (2012)Google Scholar
  14. 14.
    Welsh, T., Ashikhmin, M., Mueller, K.: Transferring color to greyscale images. ACM Trans. Graph. (TOG) 21, 277–280 (2002)CrossRefGoogle Scholar
  15. 15.
    Xia, S., Liu, J., Fang, Y., Yang, W., Guo, Z.: Robust and automatic video colorization via multiframe reordering refinement. In: IEEE International Conference on Image Processing, pp. 4017–4021. IEEE (2016)Google Scholar
  16. 16.
    Yatziv, L., Sapiro, G.: Fast image and video colorization using chrominance blending. IEEE Trans. Image Process. 15(5), 1120–1129 (2006)CrossRefGoogle Scholar
  17. 17.
    Zhang, R., Isola, P., Efros, A.A.: Colorful image colorization. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 649–666. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46487-9_40CrossRefGoogle Scholar
  18. 18.
    Dial M for murder. https://www.imdb.com/title/tt0046912/ (1954)
  19. 19.
    Et Dieu.créa la femme. https://www.imdb.com/title/tt0049189/ (1956)
  20. 20.
  21. 21.
    A streetcar named desire. https://www.imdb.com/title/tt0044081/ (1951)
  22. 22.
    Twelve angry men. https://www.imdb.com/title/tt0050083/ (1957)

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Panagiotis Kouzouglidis
    • 1
  • Giorgos Sfikas
    • 1
    • 2
    Email author
  • Christophoros Nikou
    • 1
  1. 1.Department of Computer Science and EngineeringUniversity of IoanninaIoanninaGreece
  2. 2.Information Technologies Institute, CERTHThessalonikiGreece

Personalised recommendations