Deep Fundamental Matrix Estimation

  • René RanftlEmail author
  • Vladlen Koltun
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11205)


We present an approach to robust estimation of fundamental matrices from noisy data contaminated by outliers. The problem is cast as a series of weighted homogeneous least-squares problems, where robust weights are estimated using deep networks. The presented formulation acts directly on putative correspondences and thus fits into standard 3D vision pipelines that perform feature extraction, matching, and model fitting. The approach can be trained end-to-end and yields computationally efficient robust estimators. Our experiments indicate that the presented approach is able to train robust estimators that outperform classic approaches on real data by a significant margin.

Supplementary material

474172_1_En_18_MOESM1_ESM.pdf (2 mb)
Supplementary material 1 (pdf 2088 KB)


  1. 1.
    Agrawal, P., Carreira, J., Malik, J.: Learning to see by moving. In: ICCV, pp. 37–45 (2015)Google Scholar
  2. 2.
    Andrychowicz, M., et al.: Learning to learn by gradient descent by gradient descent. In: NIPS, pp. 3981–3989 (2016)Google Scholar
  3. 3.
    Brachmann, E., et al.: DSAC - Differentiable RANSAC for camera localization. In: CVPR (2017)Google Scholar
  4. 4.
    Chin, T., Purkait, P., Eriksson, A.P., Suter, D.: Efficient globally optimal consensus maximisation with tree search. IEEE Trans. Pattern Anal. Mach. Intell. 39(4), 758–772 (2017)CrossRefGoogle Scholar
  5. 5.
    Chum, O., Matas, J.: Matching with PROSAC - progressive sample consensus. In: CVPR (2005)Google Scholar
  6. 6.
    DeTone, D., Malisiewicz, T., Rabinovich, A.: Deep image homography estimation (2016). arXiv:1606.03798
  7. 7.
    Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24(6), 381–395 (1981)MathSciNetCrossRefGoogle Scholar
  8. 8.
    Fitzgibbon, A.W.: Robust registration of 2D and 3D point sets. Image Vis. Comput. 21(13–14), 1145–1153 (2003)CrossRefGoogle Scholar
  9. 9.
    Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving? The KITTI vision benchmark suite. In: CVPR (2012)Google Scholar
  10. 10.
    Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: CVPR (2014)Google Scholar
  11. 11.
    Grabner, H., Gall, J., Gool, L.J.V.: What makes a chair a chair? In: CVPR (2011)Google Scholar
  12. 12.
    Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge University Press, Cambridge (2000)zbMATHGoogle Scholar
  13. 13.
    Hartley, R.I.: In defense of the eight-point algorithm. IEEE Trans. Pattern Anal. Mach. Intell. 19(6), 580–593 (1997)CrossRefGoogle Scholar
  14. 14.
    Hoseinnezhad, R., Bab-Hadiashar, A.: An M-estimator for high breakdown robust estimation in computer vision. Comput. Vis. Image Underst. 115(8), 1145–1156 (2011)CrossRefGoogle Scholar
  15. 15.
    Ionescu, C., Vantzos, O., Sminchisescu, C.: Matrix backpropagation for deep networks with structured layers. In: ICCV (2015)Google Scholar
  16. 16.
    Kendall, A., Cipolla, R.: Geometric loss functions for camera pose regression with deep learning. In: CVPR (2017)Google Scholar
  17. 17.
    Kendall, A., Grimes, M., Cipolla, R.: PoseNet: A convolutional network for real-time 6-DOF camera relocalization. In: ICCV (2015)Google Scholar
  18. 18.
    Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: ICLR (2015)Google Scholar
  19. 19.
    Knapitsch, A., Park, J., Zhou, Q.Y., Koltun, V.: Tanks and temples: benchmarking large-scale scene reconstruction. ACM Trans. Graph. 36(4), 78 (2017)CrossRefGoogle Scholar
  20. 20.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: NIPS (2012)Google Scholar
  21. 21.
    Lebeda, K., Matas, J., Chum, O.: Fixing the locally optimized RANSAC. In: BMVC (2012)Google Scholar
  22. 22.
    Li, H.: Consensus set maximization with guaranteed global optimality for robust geometry estimation. In: ICCV (2009)Google Scholar
  23. 23.
    Lin, T., et al.: Microsoft COCO: common objects in context. In: ECCV (2014)Google Scholar
  24. 24.
    Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: CVPR (2015)Google Scholar
  25. 25.
    Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004)MathSciNetCrossRefGoogle Scholar
  26. 26.
    Maas, A.L., Hannun, A.Y., Ng, A.Y.: Rectifier nonlinearities improve neural network acoustic models. In: ICML Workshops (2013)Google Scholar
  27. 27.
    Mur-Artal, R., Montiel, J.M.M., Tardós, J.D.: ORB-SLAM: a versatile and accurate monocular SLAM system. IEEE Trans. Robot. 31(5), 1147–1163 (2015)CrossRefGoogle Scholar
  28. 28.
    Nguyen, T., Chen, S.W., Shivakumar, S.S., Taylor, C.J., Kumar, V.: Unsupervised deep homography: A fast and robust homography estimation model (2017). arXiv:1709.03966
  29. 29.
    Qi, C.R., Su, H., Mo, K., Guibas, L.J.: PointNet: Deep learning on point sets for 3D classification and segmentation. In: CVPR (2016)Google Scholar
  30. 30.
    Raguram, R., Chum, O., Pollefeys, M., Matas, J., Frahm, J.: USAC: a universal framework for random sample consensus. IEEE Trans. Pattern Anal. Mach. Intell. 35(8), 2022–2038 (2013)CrossRefGoogle Scholar
  31. 31.
    Rocco, I., Arandjelovic, R., Sivic, J.: Convolutional neural network architecture for geometric matching. In: CVPR (2017)Google Scholar
  32. 32.
    Rousseeuw, P.J.: Least median of squares regression. J. Am. Stat. Assoc. 79(388), 871–880 (1984)MathSciNetCrossRefGoogle Scholar
  33. 33.
    Rublee, E., Rabaud, V., Konolige, K., Bradski, G.R.: ORB: An efficient alternative to SIFT or SURF. In: ICCV (2011)Google Scholar
  34. 34.
    Savinov, N., Seki, A., Ladicky, L., Sattler, T., Pollefeys, M.: Quad-networks: unsupervised learning to rank for interest point detection. In: CVPR (2017)Google Scholar
  35. 35.
    Schönberger, J.L., Frahm, J.M.: Structure-from-motion revisited. In: CVPR (2016)Google Scholar
  36. 36.
    Tennakoon, R.B., Bab-Hadiashar, A., Cao, Z., Hoseinnezhad, R., Suter, D.: Robust model fitting using higher than minimal subset sampling. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 350–362 (2016)CrossRefGoogle Scholar
  37. 37.
    Torr, P.H.S.: Bayesian model estimation and selection for epipolar geometry and generic manifold fitting. Int. J. Comput. Vis. 50(1), 35–61 (2002)CrossRefGoogle Scholar
  38. 38.
    Torr, P.H.S., Murray, D.W.: The development and comparison of robust methods for estimating the fundamental matrix. Int. J. Comput. Vis. 24(3), 271–300 (1997)CrossRefGoogle Scholar
  39. 39.
    Torr, P.H.S., Zisserman, A.: MLESAC: a new robust estimator with application to estimating image geometry. Comput. Vis. Image Underst. 78(1), 138–156 (2000)CrossRefGoogle Scholar
  40. 40.
    Ulyanov, D., Vedaldi, A., Lempitsky, V.S.: Improved texture networks: maximizing quality and diversity in feed-forward stylization and texture synthesis. In: CVPR (2017)Google Scholar
  41. 41.
    Ummenhofer, B., Zhou, H., Uhrig, J., Mayer, N., Ilg, E., Dosovitskiy, A., Brox, T.: DeMoN: Depth and motion network for learning monocular stereo. In: CVPR (2017)Google Scholar
  42. 42.
    Vongkulbhisal, J., De la Torre, F., Costeira, J.P.: Discriminative optimization: theory and applications to point cloud registration. In: CVPR (2017)Google Scholar
  43. 43.
    Wilson, K., Snavely, N.: Robust global translations with 1DSfM. In: ECCV (2014)Google Scholar
  44. 44.
    Yang, J., Li, H., Jia, Y.: Optimal essential matrix estimation via inlier-set maximization. In: ECCV (2014)Google Scholar
  45. 45.
    Yi, K.M., Trulls, E., Lepetit, V., Fua, P.: LIFT: Learned invariant feature transform. In: ECCV (2016)Google Scholar
  46. 46.
    Yi, K.M., Trulls, E., Ono, Y., Lepetit, V., Salzmann, M., Fua, P.: Learning to find good correspondences. In: CVPR (2018)Google Scholar
  47. 47.
    Zach, C.: Robust bundle adjustment revisited. In: ECCV (2014)Google Scholar
  48. 48.
    Zaheer, M., Kottur, S., Ravanbakhsh, S., Póczos, B., Salakhutdinov, R., Smola, A.J.: Deep sets. In: NIPS (2017)Google Scholar
  49. 49.
    Zhang, Z.: Determining the epipolar geometry and its uncertainty: a review. Int. J. Comput. Vis. 27(2), 161–195 (1998)CrossRefGoogle Scholar
  50. 50.
    Zhou, Q., Park, J., Koltun, V.: Fast global registration. In: ECCV (2016)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.Intel LabsMunichGermany

Personalised recommendations