Abstract
We propose a framework for temporally consistent video completion. To this end we generalize the exemplar-based inpainting method of Criminisi et al. [7] to video inpainting. Specifically we address two important issues: Firstly, we propose a color and optical flow inpainting to ensure temporal consistency of inpainting even for complex motion of foreground and background. Secondly, rather than requiring the user to hand-label the inpainting region in every single image, we propose a flow-based propagation of user scribbles from the first to subsequent video frames which drastically reduces the user input. Experimental comparisons to state-of-the-art video completion methods demonstrate the benefits of the proposed approach.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Ashikhmin, M.: Synthesizing natural textures. In: Proceedings of the 2001 Symposium on Interactive 3D Graphics, pp. 217–226. ACM (2001)
Barnes, C., Shechtman, E., Finkelstein, A., Goldman, D.B.: Patchmatch: a randomized correspondence algorithm for structural image editing. ACM Trans. Graph. 28(3), Article 24, 1–11 (2009)
Bertalmio, M., Bertozzi, A.L., Sapiro, G.: Navier-stokes, fluid dynamics, and image and video inpainting. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), pp. 355–362 (2001)
Bertalmio, M., Sapiro, G., Caselles, V., Ballester, C.: Image inpainting. In: Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, pp. 417–424. ACM Press/Addison-Wesley Publishing Co. (2000)
Brox, T., Bruhn, A., Papenberg, N., Weickert, J.: High accuracy optical flow estimation based on a theory for warping. In: Pajdla, T., Matas, J.G. (eds.) ECCV 2004. LNCS, vol. 3024, pp. 25–36. Springer, Heidelberg (2004)
Criminisi, A., Perez, P., Toyama, K.: Object removal by exemplar-based inpainting. In: International Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 721–728, June 2003
Criminisi, A., Perez, P., Toyama, K.: Region filling and object removal by exemplar-based image inpainting. IEEE Trans. Image Process. 13(9), 1200–1212 (2004)
Efros, A.A., Freeman, W.T.: Image quilting for texture synthesis and transfer. In: Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, pp. 341–346. ACM (2001)
Efros, A.A., Leung, T.K.: Texture synthesis by non-parametric sampling. In: The Proceedings of the Seventh IEEE International Conference on Computer Vision, vol. 2, pp. 1033–1038 (1999)
Granados, M., Kim, K.I., Tompkin, J., Kautz, J., Theobalt, C.: Background inpainting for videos with dynamic objects and a free-moving camera. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part I. LNCS, vol. 7572, pp. 682–695. Springer, Heidelberg (2012)
Granados, M., Tompkin, J., Kim, K.I., Grau, O., Kautz, J., Theobalt, C.: How not to be seen - object removal from videos of crowded scenes. Comput. Graph. Forum 31(2), 219–228 (2012)
Masnou, S.: Disocclusion: a variational approach using level lines. IEEE Trans. Image Process. 11(2), 68–76 (2002)
Masnou, S., Morel, J.M.: Level lines based disocclusion. In: International Conference on Image Processing, vol. 3, pp. 259–263 (1998)
Newson, A., Almansa, A., Fradet, M., Gousseau, Y., Pérez, P.: Towards fast, generic video inpainting. In: Proceedings of the 10th European Conference on Visual Media Production, CVMP ’13, pp. 1–8. ACM, New York (2013)
Newson, A., Almansa, A., Fradet, M., Gousseau, Y., Pérez, P.: Video inpainting of complex scenes, January 2014. http://hal.archives-ouvertes.fr/hal-00937795
Nieuwenhuis, C., Cremers, D.: Spatially varying color distributions for interactive multi-label segmentation. IEEE Trans. Patt. Anal. Mach. Intell. 35(5), 1234–1247 (2013)
Patwardhan, K., Sapiro, G., Bertalmio, M.: Video inpainting of occluding and occluded objects. In: IEEE International Conference on Image Processing, vol. 2, pp. 69–72 (2005)
Patwardhan, K.A., Sapiro, G., Bertalmo, M.: Video inpainting under constrained camera motion. IEEE Trans. Image Process. 16(2), 545–553 (2007)
Sethian, J.A.: A fast marching level set method for monotonically advancing fronts. Proc. Natl. Acad. Sci. 93(4), 1591–1595 (1996)
Shih, T., Tang, N., Hwang, J.N.: Exemplar-based video inpainting without ghost shadow artifacts by maintaining temporal continuity. IEEE Trans. Circuits Syst. Video Technol. 19(3), 347–360 (2009)
Telea, A.: An image inpainting technique based on the fast marching method. J. Graph. Tools 9(1), 23–34 (2004)
Tsitsiklis, J.N.: Efficient algorithms for globally optimal trajectories. IEEE Trans. Autom. Control 40(9), 1528–1538 (1995)
Wang, J., Lu, K., Pan, D., He, N., kun Bao, B.: Robust object removal with an exemplar-based image inpainting approach. Neurocomputing 123, 150–155 (2014), contains Special issue articles: Advances in Pattern Recognition Applications and Methods
Werlberger, M.: Convex approaches for high performance video processing. Ph.D. thesis, Institute for Computer Graphics and Vision, Graz University of Technology, Graz, Austria, June 2012
Wexler, Y., Shechtman, E., Irani, M.: Space-time video completion. In: International Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 120–127, June 2004
Wexler, Y., Shechtman, E., Irani, M.: Space-time completion of video. IEEE Trans. Patt. Anal. Mach. Intell. 29(3), 463–476 (2007)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this paper
Cite this paper
Strobel, M., Diebold, J., Cremers, D. (2014). Flow and Color Inpainting for Video Completion. In: Jiang, X., Hornegger, J., Koch, R. (eds) Pattern Recognition. GCPR 2014. Lecture Notes in Computer Science(), vol 8753. Springer, Cham. https://doi.org/10.1007/978-3-319-11752-2_23
Download citation
DOI: https://doi.org/10.1007/978-3-319-11752-2_23
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-11751-5
Online ISBN: 978-3-319-11752-2
eBook Packages: Computer ScienceComputer Science (R0)