Skip to main content
SpringerLink
Log in

Automatic Video Colorization Using 3D Conditional Generative Adversarial Networks

  • Conference paper
  • First Online:
Advances in Visual Computing (ISVC 2019)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 11844))

Included in the following conference series:

Abstract

In this work, we present a method for automatic colorization of grayscale videos. The core of the method is a Generative Adversarial Network that is trained and tested on sequences of frames in a sliding window manner. Network convolutional and deconvolutional layers are three-dimensional, with frame height, width and time as the dimensions taken into account. Multiple chrominance estimates per frame are aggregated and combined with available luminance information to recreate a colored sequence. Colorization trials are run successfully on a dataset of old black-and-white films. The usefulness of our method is also validated with numerical results, computed with a newly proposed metric that measures colorization consistency over a frame sequence.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Other variants of a cGAN are possible; for example, a noise variable z could be added to produce a non-deterministic output [5]. We employ a deterministic cGAN variant in this work.

  2. 2.

    \(\chi _i\) denotes the \(i^{th}\) colorization estimate for a frame. y denotes a colorization estimate for a sequence of C frames.

  3. 3.

    http://www.cs.uoi.gr/~sfikas/video_colorization.

References

  1. Ben-Zrihem, N., Zelnik-Manor, L.: Approximate nearest neighbor fields in video. In: IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5233–5242 (2015)

    Google Scholar 

  2. Chintala, S., Denton, E., Arjovsky, M., Mathieu, M.: How to train a GAN? Tips and tricks to make GANs work (2016). http://github.com/soumith/ganhacks. Accessed 25 January 2018

  3. Daskalakis, C., Ilyas, A., Syrgkanis, V., Zeng, H.: Training GANs with optimism. CoRR abs/1711.00141 (2017). http://arxiv.org/abs/1711.00141

  4. Goodfellow, I., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems (NIPS), pp. 2672–2680 (2014)

    Google Scholar 

  5. Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. arXiv preprint arXiv:1611.07004 (2016)

  6. Juliani, A.: Pix2Pix-Film (2017). http://github.com/awjuliani/Pix2Pix-Film. Accessed 2 January 2018

  7. Levin, A., Lischinski, D., Weiss, Y.: Colorization using optimization. ACM Trans. Graph. (TOG) 23, 689–694 (2004)

    Article  Google Scholar 

  8. Meyer, S., Cornillère, V., Djelouah, A., Schroers, C., Gross, M.: Deep video color propagation. arXiv preprint arXiv:1808.03232 (2018)

  9. Otani, M., Hioki, H.: Video colorization based on optical flow and edge-oriented color propagation. In: Computational Imaging XII. vol. 9020, p. 902002. International Society for Optics and Photonics (2014)

    Google Scholar 

  10. Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28

    Chapter  Google Scholar 

  11. Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., Chen, X.: Improved techniques for training GANs. In: Advances in neural information processing systems (NIPS), pp. 2234–2242 (2016)

    Google Scholar 

  12. Sheng, B., Sun, H., Magnor, M., Li, P.: Video colorization using parallel optimization in feature space. IEEE Trans. Circuits Syst. Video Technol. 24(3), 407–417 (2014)

    Article  Google Scholar 

  13. Veeravasarapu, V.R., Sivaswamy, J.: Fast and fully automated video colorization. In: 2012 International Conference on Signal Processing and Communications (SPCOM), pp. 1–5. IEEE (2012)

    Google Scholar 

  14. Welsh, T., Ashikhmin, M., Mueller, K.: Transferring color to greyscale images. ACM Trans. Graph. (TOG) 21, 277–280 (2002)

    Article  Google Scholar 

  15. Xia, S., Liu, J., Fang, Y., Yang, W., Guo, Z.: Robust and automatic video colorization via multiframe reordering refinement. In: IEEE International Conference on Image Processing, pp. 4017–4021. IEEE (2016)

    Google Scholar 

  16. Yatziv, L., Sapiro, G.: Fast image and video colorization using chrominance blending. IEEE Trans. Image Process. 15(5), 1120–1129 (2006)

    Article  Google Scholar 

  17. Zhang, R., Isola, P., Efros, A.A.: Colorful image colorization. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 649–666. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46487-9_40

    Chapter  Google Scholar 

  18. Dial M for murder. https://www.imdb.com/title/tt0046912/ (1954)

  19. Et Dieu.créa la femme. https://www.imdb.com/title/tt0049189/ (1956)

  20. Tzéni, tzéni. https://www.imdb.com/title/tt0145006/ (1966)

  21. A streetcar named desire. https://www.imdb.com/title/tt0044081/ (1951)

  22. Twelve angry men. https://www.imdb.com/title/tt0050083/ (1957)

Download references

Acknowledgements

We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan XP GPU used for this research.

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, University of Ioannina, 45110, Ioannina, Greece

    Panagiotis Kouzouglidis, Giorgos Sfikas & Christophoros Nikou

  2. Information Technologies Institute, CERTH, 57001, Thessaloniki, Greece

    Giorgos Sfikas

Authors
  1. Panagiotis Kouzouglidis
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Giorgos Sfikas
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Christophoros Nikou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Giorgos Sfikas .

Editor information

Editors and Affiliations

  1. University of Nevada, Reno, NV, USA

    George Bebis

  2. NASA Ames Research Center, Moffett Field, CA, USA

    Richard Boyle

  3. University of Nevada, Reno, NV, USA

    Bahram Parvin

  4. Desert Research Institute, Reno, NV, USA

    Darko Koracin

  5. Lawrence Berkeley National Laboratory, Berkeley, CA, USA

    Daniela Ushizima

  6. Latent AI, Palo Alto, CA, USA

    Sek Chai

  7. Texas A&M University, College Station, TX, USA

    Shinjiro Sueda

  8. Louisiana State University, Baton Rouge, LA, USA

    Xin Lin

  9. University of North Carolina at Charlotte, Charlotte, NC, USA

    Aidong Lu

  10. École Polytechnique Fédérale de Lausanne, Lausanne, Switzerland

    Daniel Thalmann

  11. Notre Dame University, Notre Dame, IN, USA

    Chaoli Wang

  12. Bosch Research North America, Palo Alto, CA, USA

    Panpan Xu

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Kouzouglidis, P., Sfikas, G., Nikou, C. (2019). Automatic Video Colorization Using 3D Conditional Generative Adversarial Networks. In: Bebis, G., et al. Advances in Visual Computing. ISVC 2019. Lecture Notes in Computer Science(), vol 11844. Springer, Cham. https://doi.org/10.1007/978-3-030-33720-9_16

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-33720-9_16

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-33719-3

  • Online ISBN: 978-3-030-33720-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation