Advertisement

Learning to Dodge A Bullet: Concyclic View Morphing via Deep Learning

  • Shi Jin
  • Ruiynag Liu
  • Yu Ji
  • Jinwei Ye
  • Jingyi Yu
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11218)

Abstract

The bullet-time effect, presented in feature film “The Matrix”, has been widely adopted in feature films and TV commercials to create an amazing stopping-time illusion. Producing such visual effects, however, typically requires using a large number of cameras/images surrounding the subject. In this paper, we present a learning-based solution that is capable of producing the bullet-time effect from only a small set of images. Specifically, we present a view morphing framework that can synthesize smooth and realistic transitions along a circular view path using as few as three reference images. We apply a novel cyclic rectification technique to align the reference images onto a common circle and then feed the rectified results into a deep network to predict its motion field and per-pixel visibility for new view interpolation. Comprehensive experiments on synthetic and real data show that our new framework outperforms the state-of-the-art and provides an inexpensive and practical solution for producing the bullet-time effects.

Keywords

Bullet-time effect Image-based rendering View morphing Convolutional neural network (CNN) 

Supplementary material

474202_1_En_14_MOESM2_ESM.pdf (4.5 mb)
Supplementary material 2 (pdf 4658 KB)

References

  1. 1.
    Carranza, J., Theobalt, C., Magnor, M.A., Seidel, H.P.: Free-viewpoint video of human actors. ACM Trans. Graph. 22(3), 569–577 (2003)CrossRefGoogle Scholar
  2. 2.
    Zitnick, C.L., Kang, S.B., Uyttendaele, M., Winder, S., Szeliski, R.: High-quality video view interpolation using a layered representation. ACM Trans. Graph. 23(3), 600–608 (2004)CrossRefGoogle Scholar
  3. 3.
    Liao, J., Lima, R.S., Nehab, D., Hoppe, H., Sander, P.V., Yu, J.: Automating image morphing using structural similarity on a halfway domain. ACM Trans. Graph. 33(5), 168:1–168:12 (2014)CrossRefGoogle Scholar
  4. 4.
    Linz, C., Lipski, C., Rogge, L., Theobalt, C., Magnor, M.: Space-time visual effects as a post-production process. In: Proceedings of the 1st International Workshop on 3D Video Processing. ACM (2010)Google Scholar
  5. 5.
    Seitz, S.M., Dyer, C.R.: View morphing. In: Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques. In: SIGGRAPH 1996, pp. 21–30. ACM (1996)Google Scholar
  6. 6.
    Ji, D., Kwon, J., McFarland, M., Savarese, S.: Deep view morphing. In: IEEE Conference on Computer Vision and Pattern Recognition (2017)Google Scholar
  7. 7.
    Park, E., Yang, J., Yumer, E., Ceylan, D., Berg, A.C.: Transformation-grounded image generation network for novel 3D view synthesis. In: IEEE Conference on Computer Vision and Pattern Recognition (2017)Google Scholar
  8. 8.
    Zhou, T., Tulsiani, S., Sun, W., Malik, J., Efros, A.A.: View synthesis by appearance flow. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9908, pp. 286–301. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46493-0_18CrossRefGoogle Scholar
  9. 9.
    Varol, G., et al.: Learning from Synthetic Humans. In: The IEEE Conference on Computer Vision and Pattern Recognition (2017)Google Scholar
  10. 10.
    Chang, A.X., et al.: ShapeNet: an Information-Rich 3D Model Repository. Technical report arXiv:1512.03012 (2015)
  11. 11.
    Levoy, M., Hanrahan, P.: Light field rendering. In: Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 1996, pp. 31–42. ACM (1996)Google Scholar
  12. 12.
    Gortler, S.J., Grzeszczuk, R., Szeliski, R., Cohen, M.F.: The lumigraph. In: Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques. In: SIGGRAPH 1996, pp. 43–54. ACM (1996)Google Scholar
  13. 13.
    Penner, E., Zhang, L.: Soft 3D reconstruction for view synthesis. ACM Trans. Graph. 36(6), 235:1–235:11 (2017)CrossRefGoogle Scholar
  14. 14.
    Rematas, K., Nguyen, C.H., Ritschel, T., Fritz, M., Tuytelaars, T.: Novel views of objects from a single image. IEEE Trans. Pattern Anal. Mach. Intell. 39(8), 1576–1590 (2017)CrossRefGoogle Scholar
  15. 15.
    Lipski, C., Linz, C., Berger, K., Sellent, A., Magnor, M.: Virtual video camera: image-based viewpoint navigation through space and time. In: Computer Graphics Forum, pp. 2555–2568. Blackwell Publishing Ltd., Oxford (2010)CrossRefGoogle Scholar
  16. 16.
    Ballan, L., Brostow, G.J., Puwein, J., Pollefeys, M.: Unstructured video-based rendering: Interactive exploration of casually captured videos. ACM Trans. Graph. 29(4), 87:1–87:11 (2010)CrossRefGoogle Scholar
  17. 17.
    Zhang, Z., Wang, L., Guo, B., Shum, H.Y.: Feature-based light field morphing. ACM Trans. Graph. 21(3), 457–464 (2002)CrossRefGoogle Scholar
  18. 18.
    Beier, T., Neely, S.: Feature-based image metamorphosis. In: Proceedings of the 19th Annual Conference on Computer Graphics and Interactive Techniques. In: SIGGRAPH 1992, pp. 35–42 (1992)Google Scholar
  19. 19.
    Lee, S., Wolberg, G., Shin, S.Y.: Polymorph: morphing among multiple images. IEEE Comput. Graph. Appl. 18(1), 58–71 (1998)CrossRefGoogle Scholar
  20. 20.
    Quenot, G.M.: Image matching using dynamic programming: application to stereovision and image interpolation. In: Image Communication (1996)Google Scholar
  21. 21.
    Chaurasia, G., Sorkine-Hornung, O., Drettakis, G.: Silhouette-aware warping for image-based rendering. In: Computer Graphics Forum (Proceedings of the Eurographics Symposium on Rendering), vol. 30, no. 4. Blackwell Publishing Ltd., Oxford (2011)CrossRefGoogle Scholar
  22. 22.
    Germann, M., Popa, T., Keiser, R., Ziegler, R., Gross, M.: Novel-view synthesis of outdoor sport events using an adaptive view-dependent geometry. Comput. Graph. Forum 31, 325–333 (2012)CrossRefGoogle Scholar
  23. 23.
    Mahajan, D., Huang, F.C., Matusik, W., Ramamoorthi, R., Belhumeur, P.: Moving gradients: a path-based method for plausible image interpolation. ACM Trans. Graph. 28(3), 42:1–42:11 (2009)CrossRefGoogle Scholar
  24. 24.
    Dosovitskiy, A., Springenberg, J.T., Brox, T.: Learning to generate chairs with convolutional neural networks. In: IEEE Conference on Computer Vision and Pattern Recognition (2015)Google Scholar
  25. 25.
    Tatarchenko, M., Dosovitskiy, A., Brox, T.: Multi-view 3D models from single images with a convolutional network. In: European Conference on Computer Vision (2016)Google Scholar
  26. 26.
    Niklaus, S., Mai, L., Liu, F.: Video frame interpolation via adaptive convolution. In: IEEE Conference on Computer Vision and Pattern Recognition (2017)Google Scholar
  27. 27.
    Niklaus, S., Mai, L., Liu, F.: Video frame interpolation via adaptive separable convolution. In: IEEE International Conference on Computer Vision (2017)Google Scholar
  28. 28.
    Jaderberg, M., Simonyan, K., Zisserman, A., Kavukcuoglu, K.: Spatial transformer networks. In: Proceedings of the 28th International Conference on Neural Information Processing Systems, pp. 2017–2025 (2015)Google Scholar
  29. 29.
    Flynn, J., Neulander, I., Philbin, J., Snavely, N.: Deep stereo: learning to predict new views from the world’s imagery. In: IEEE Conference on Computer Vision and Pattern Recognition (2016)Google Scholar
  30. 30.
    Kalantari, N.K., Wang, T.C., Ramamoorthi, R.: Learning-based view synthesis for light field cameras. ACM Trans. Graph. 35(6), 193:1–193:10 (2016)CrossRefGoogle Scholar
  31. 31.
    Ilg, E., Mayer, N., Saikia, T., Keuper, M., Dosovitskiy, A., Brox, T.: FlowNet 2.0: evolution of optical flow estimation with deep networks. In: IEEE Conference on Computer Vision and Pattern Recognition (2017)Google Scholar
  32. 32.
    Newell, A., Yang, K., Deng, J.: Stacked hourglass networks for human pose estimation. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9912, pp. 483–499. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46484-8_29CrossRefGoogle Scholar
  33. 33.
    Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., Black, M.J.: SMPL: a skinned multi-person linear model. ACM Trans. Graph. 34(6), 248:1–248:16 (2015). (Proc. SIGGRAPH Asia)CrossRefGoogle Scholar
  34. 34.
    Rematas, K., Ritschel, T., Fritz, M., Gavves, E., Tuytelaars, T.: Deep reflectance maps. In: IEEE Conference on Computer Vision and Pattern Recognition (2016)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Shi Jin
    • 1
    • 3
  • Ruiynag Liu
    • 1
    • 3
  • Yu Ji
    • 2
  • Jinwei Ye
    • 3
  • Jingyi Yu
    • 1
    • 2
  1. 1.ShanghaiTech UniversityShanghaiChina
  2. 2.Plex-VRBaton RougeUSA
  3. 3.Louisiana State UniversityBaton RougeUSA

Personalised recommendations