Joint Caption Detection and Inpainting Using Generative Network

  • Vismay Patel
  • Anubha PandeyEmail author
Conference paper
Part of the The Springer Series on Challenges in Machine Learning book series (SSCML)


Video decaptioning is the task of removing overlayed text in the video sequences and filling it with coherent and semantically meaningful content. In this work, we propose a generative CNN to do caption detection and decaptioning task jointly in an end-to-end fashion. Our network does frame level decaptioning and thus the same network can be adapted to do image decaptioning or other similar tasks. Our network is capable of inpainting video frames that contain text overlays in various size, background, color, and location. All the experiments were performed on the dataset provided in the ECCV’18 Satellite Workshop Chalearn LAP Inpainting Competition Track 2 - Video decaptioning. We also secured the third rank in the competition. Most of our work is inspired by previous work on image generation and image inpainting.


  1. 1.
    Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Advances in neural information processing systems, pp. 2672–2680 (2014)Google Scholar
  2. 2.
    Iizuka, S., Simo-Serra, E., Ishikawa, H.: Globally and locally consistent image completion. ACM Transactions on Graphics (TOG) 36(4), 107 (2017)Google Scholar
  3. 3.
    Jia, Y.T., Hu, S.M., Martin, R.R.: Video completion using tracking and fragment merging. The Visual Computer 21(8–10), 601–610 (2005)CrossRefGoogle Scholar
  4. 4.
    Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: European Conference on Computer Vision, pp. 694–711. Springer (2016)Google Scholar
  5. 5.
    Newson, A., Almansa, A., Fradet, M., Gousseau, Y., Pérez, P.: Video inpainting of complex scenes. SIAM Journal on Imaging Sciences 7(4), 1993–2019 (2014)MathSciNetCrossRefGoogle Scholar
  6. 6.
    Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., Efros, A.A.: Context encoders: Feature learning by inpainting. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2536–2544 (2016)Google Scholar
  7. 7.
    Shen, Y., Lu, F., Cao, X., Foroosh, H.: Video completion for perspective camera under constrained motion. In: null, pp. 63–66. IEEE (2006)Google Scholar
  8. 8.
    Wexler, Y., Shechtman, E., Irani, M.: Space-time video completion. In: null, pp. 120–127. IEEE (2004)Google Scholar
  9. 9.
    Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.S.: Generative image inpainting with contextual attention. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5505–5514 (2018)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Indian Institute of Technology MadrasChennaiIndia

Personalised recommendations