Abstract
In this work, we present a method for automatic colorization of grayscale videos. The core of the method is a Generative Adversarial Network that is trained and tested on sequences of frames in a sliding window manner. Network convolutional and deconvolutional layers are three-dimensional, with frame height, width and time as the dimensions taken into account. Multiple chrominance estimates per frame are aggregated and combined with available luminance information to recreate a colored sequence. Colorization trials are run successfully on a dataset of old black-and-white films. The usefulness of our method is also validated with numerical results, computed with a newly proposed metric that measures colorization consistency over a frame sequence.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Other variants of a cGAN are possible; for example, a noise variable z could be added to produce a non-deterministic output [5]. We employ a deterministic cGAN variant in this work.
- 2.
\(\chi _i\) denotes the \(i^{th}\) colorization estimate for a frame. y denotes a colorization estimate for a sequence of C frames.
- 3.
References
Ben-Zrihem, N., Zelnik-Manor, L.: Approximate nearest neighbor fields in video. In: IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5233–5242 (2015)
Chintala, S., Denton, E., Arjovsky, M., Mathieu, M.: How to train a GAN? Tips and tricks to make GANs work (2016). http://github.com/soumith/ganhacks. Accessed 25 January 2018
Daskalakis, C., Ilyas, A., Syrgkanis, V., Zeng, H.: Training GANs with optimism. CoRR abs/1711.00141 (2017). http://arxiv.org/abs/1711.00141
Goodfellow, I., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems (NIPS), pp. 2672–2680 (2014)
Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. arXiv preprint arXiv:1611.07004 (2016)
Juliani, A.: Pix2Pix-Film (2017). http://github.com/awjuliani/Pix2Pix-Film. Accessed 2 January 2018
Levin, A., Lischinski, D., Weiss, Y.: Colorization using optimization. ACM Trans. Graph. (TOG) 23, 689–694 (2004)
Meyer, S., Cornillère, V., Djelouah, A., Schroers, C., Gross, M.: Deep video color propagation. arXiv preprint arXiv:1808.03232 (2018)
Otani, M., Hioki, H.: Video colorization based on optical flow and edge-oriented color propagation. In: Computational Imaging XII. vol. 9020, p. 902002. International Society for Optics and Photonics (2014)
Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28
Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., Chen, X.: Improved techniques for training GANs. In: Advances in neural information processing systems (NIPS), pp. 2234–2242 (2016)
Sheng, B., Sun, H., Magnor, M., Li, P.: Video colorization using parallel optimization in feature space. IEEE Trans. Circuits Syst. Video Technol. 24(3), 407–417 (2014)
Veeravasarapu, V.R., Sivaswamy, J.: Fast and fully automated video colorization. In: 2012 International Conference on Signal Processing and Communications (SPCOM), pp. 1–5. IEEE (2012)
Welsh, T., Ashikhmin, M., Mueller, K.: Transferring color to greyscale images. ACM Trans. Graph. (TOG) 21, 277–280 (2002)
Xia, S., Liu, J., Fang, Y., Yang, W., Guo, Z.: Robust and automatic video colorization via multiframe reordering refinement. In: IEEE International Conference on Image Processing, pp. 4017–4021. IEEE (2016)
Yatziv, L., Sapiro, G.: Fast image and video colorization using chrominance blending. IEEE Trans. Image Process. 15(5), 1120–1129 (2006)
Zhang, R., Isola, P., Efros, A.A.: Colorful image colorization. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 649–666. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46487-9_40
Dial M for murder. https://www.imdb.com/title/tt0046912/ (1954)
Et Dieu.créa la femme. https://www.imdb.com/title/tt0049189/ (1956)
Tzéni, tzéni. https://www.imdb.com/title/tt0145006/ (1966)
A streetcar named desire. https://www.imdb.com/title/tt0044081/ (1951)
Twelve angry men. https://www.imdb.com/title/tt0050083/ (1957)
Acknowledgements
We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan XP GPU used for this research.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Kouzouglidis, P., Sfikas, G., Nikou, C. (2019). Automatic Video Colorization Using 3D Conditional Generative Adversarial Networks. In: Bebis, G., et al. Advances in Visual Computing. ISVC 2019. Lecture Notes in Computer Science(), vol 11844. Springer, Cham. https://doi.org/10.1007/978-3-030-33720-9_16
Download citation
DOI: https://doi.org/10.1007/978-3-030-33720-9_16
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-33719-3
Online ISBN: 978-3-030-33720-9
eBook Packages: Computer ScienceComputer Science (R0)