Advertisement

Stereo Computation for a Single Mixture Image

  • Yiran Zhong
  • Yuchao Dai
  • Hongdong Li
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11213)

Abstract

This paper proposes an original problem of stereo computation from a single mixture image – a challenging problem that had not been researched before. The goal is to separate (i.e., unmix) a single mixture image into two constitute image layers, such that the two layers form a left-right stereo image pair, from which a valid disparity map can be recovered. This is a severely illposed problem, from one input image one effectively aims to recover three (i.e., left image, right image and a disparity map). In this work we give a novel deep-learning based solution, by jointly solving the two subtasks of image layer separation as well as stereo matching. Training our deep net is a simple task, as it does not need to have disparity maps. Extensive experiments demonstrate the efficacy of our method.

Keywords

Stereo computation Image separation Anaglyph Monocular depth estimation Double vision 

Notes

Acknowledgements

Y. Zhong’s PhD scholarship is funded by Data61. Y. Dai is supported in part by National 1000 Young Talents Plan of China, Natural Science Foundation of China (61420106007, 61671387), and ARC grant (DE140100180). H. Li’s work is funded in part by ACRV (CE140100016). The authors are very grateful to NVIDIA’s generous gift of GPUs to ANU used in this research.

Supplementary material

474192_1_En_27_MOESM1_ESM.pdf (45.3 mb)
Supplementary material 1 (pdf 46379 KB)

References

  1. 1.
    Garg, R., Vijay Kumar, B.G., Carneiro, G., Reid, I.: Unsupervised CNN for single view depth estimation: geometry to the rescue. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9912, pp. 740–756. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46484-8_45CrossRefGoogle Scholar
  2. 2.
    Godard, C., Mac Aodha, O., Brostow, G.J.: Unsupervised monocular depth estimation with left-right consistency. In: Proceeding IEEE Conference Computer Vision Pattern Recognition (2017)Google Scholar
  3. 3.
    Li, B., Shen, C., Dai, Y., van den Hengel, A., He, M.: Depth and surface normal estimation from monocular images using regression on deep features and hierarchical CRFs. In: Proceedings IEEE Conference Computer Vision Pattern Recognition, June 2015Google Scholar
  4. 4.
    Bronstein, A.M., Bronstein, M.M., Zibulevsky, M., Zeevi, Y.Y.: Sparse ICA for blind separation of transmitted and reflected images. Intl J. Imaging Sci. Technol. 15(1), 84–91 (2005)CrossRefGoogle Scholar
  5. 5.
    Yang, J., Li, H., Dai, Y., Tan, R.T.: Robust optical flow estimation of double-layer images under transparency or reflection. In: Proceedings IEEE Conference Computer Vision Pattern Recognition (June 2016)Google Scholar
  6. 6.
    Li, Z., Tan, P., Tan, R.T., Zou, D., Zhou, S.Z., Cheong, L.F.: Simultaneous video defogging and stereo reconstruction. In: Proceedings IEEE Conference Computer Vision Pattern Recognition (June 2015) 4988–4997Google Scholar
  7. 7.
    Levin, A., Lischinski, D., Weiss, Y.: A closed-form solution to natural image matting. IEEE Trans. Pattern Anal. Mach. Intell. 30(2), 228–242 (2008)CrossRefGoogle Scholar
  8. 8.
    Williem, W., Raskar, R., Park, I.K.: Depth map estimation and colorization of anaglyph images using local color prior and reverse intensity distribution. In: Proceedings IEEE International Conference Computer Vision, pp. 3460–3468 (Dec 2015)Google Scholar
  9. 9.
    Joulin, A., Kang, S.B.: Recovering stereo pairs from anaglyphs. In: Proceedings IEEE Conference Computer Vision Pattern Recognition, pp. 289–296 (2013)Google Scholar
  10. 10.
    Li, Y., Brown, M.S.: Single image layer separation using relative smoothness. In: Proceedings IEEE Conference Computer Vision Pattern Recognition (June 2014)Google Scholar
  11. 11.
    Fan, Q., Yang, J., Hua, G., Chen, B., Wipf, D.: A generic deep architecture for single image reflection removal and image smoothing. In: Proceedings IEEE International Conference Computer Vision (2017)Google Scholar
  12. 12.
    Liu, C., Yuen, J., Torralba, A.: Sift flow: dense correspondence across scenes and its applications. IEEE Trans. Pattern Anal. Mach. Intell. 33(5), 978–994 (2011)CrossRefGoogle Scholar
  13. 13.
    Xie, J., Girshick, R., Farhadi, A.: Deep3D: fully automatic 2D-to-3D video conversion with deep convolutional neural networks. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9908, pp. 842–857. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46493-0_51CrossRefGoogle Scholar
  14. 14.
    Iizuka, S., Simo-Serra, E., Ishikawa, H.: Let there be color!: Joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification. ACM Trans. Graph. 35(4), 110:1–110:11 (2016)CrossRefGoogle Scholar
  15. 15.
    Zhang, R., Isola, P., Efros, A.A.: Colorful image colorization. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 649–666. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46487-9_40CrossRefGoogle Scholar
  16. 16.
    Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: Proceedings IEEE Conference Computer Vision Pattern Recognition (2017)Google Scholar
  17. 17.
    Levin, A., Weiss, Y.: User assisted separation of reflections from a single image using a sparsity prior. IEEE Trans. Pattern Anal. Mach. Intell. 29(9), 1647–1654 (2007)CrossRefGoogle Scholar
  18. 18.
    Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015).  https://doi.org/10.1007/978-3-319-24574-4_28CrossRefGoogle Scholar
  19. 19.
    Zhong, Y., Dai, Y., Li, H.: Self-supervised learning for stereo matching with self-improving ability (2017). arXiv:1709.00930
  20. 20.
    Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Proc. 13(4), 600–612 (2004)CrossRefGoogle Scholar
  21. 21.
    Jaderberg, M., Simonyan, K., Zisserman, A., kavukcuoglu, k.: Spatial transformer networks. In: Proceedings Advances Neural Information Processing System, pp. 2017–2025 (2015)Google Scholar
  22. 22.
    Abadi, M., Agarwal, A., et al., P.B.: Tensorflow: Large-scale machine learning on heterogeneous distributed systems. CoRR arXiv:abs/1603.04467 (2016)
  23. 23.
    Tieleman, T., Hinton, G.: Lecture 6.5–RmsProp: divide the gradient by a running average of its recent magnitude. In: COURSERA: Neural Networks for Machine Learning (2012)Google Scholar
  24. 24.
    Scharstein, D., Szeliski, R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. J. Comp. Vis. 47(1–3), 7–42 (2002)CrossRefGoogle Scholar
  25. 25.
    Menze, M., Geiger, A.: Object scene flow for autonomous vehicles. In: Proceedings IEEE Conference Computer Vision Pattern Recognition (2015)Google Scholar
  26. 26.
    Li, Y., Brown, M.S.: Single image layer separation using relative smoothness. In: Proceedings IEEE Conference Computer Vision Pattern Recognition, pp. 2752–2759 (2014)Google Scholar
  27. 27.
    Eigen, D., Puhrsch, C., Fergus, R.: Depth map prediction from a single image using a multi-scale deep network. In: Proceedings Advances Neural Information Processing System (2014)Google Scholar
  28. 28.
    Li, B., Dai, Y., He, M.: Monocular depth estimation with hierarchical fusion of dilated CNNS and soft-weighted-sum inference. Pattern Recogn. 83, 328–339 (2018)CrossRefGoogle Scholar
  29. 29.
    Zhou, T., Brown, M., Snavely, N., Lowe, D.G.: Unsupervised learning of depth and ego-motion from video. In: Proceedings IEEE Conference Computer Vision Pattern Recognition (2017)Google Scholar
  30. 30.
    Yang, J., Ye, X., Li, K., Hou, C., Wang, Y.: Color-guided depth recovery from RGB-D data using an adaptive autoregressive model. In: IEEE Transactions Image Proceedings, vol. 23, pp. 3443-3458 (Aug 2014)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.Australian National UniversityCanberraAustralia
  2. 2.Northwestern Polytechnical UniversityXi’anChina
  3. 3.Data61 CSIROCanberraAustralia
  4. 4.Australian Centre for Robotic VisionCanberraAustralia

Personalised recommendations