Abstract
Bundle adjustment (BA) is the problem of refining a visual reconstruction to produce jointly optimal 3D structure and viewing parameter (camera pose and or calibration) estimates, and it is almost always used as the last step of feature-based 3D reconstruction algorithm. Generally, the result of Structure from Motion (SFM) mainly relies on the quality of BA. The problem of BA is often formulated as a nonlinear least squares problem, where the data arises from keypoints matching. For 3D reconstruction, mismatched keypoints may cause serious problems, even a single mismatch will affect the entire reconstruction. Therefore, to further impove the robustness of BA algorithm is very necessary. In this paper, we propose a robust Bundle Adjustment (RBA) algorithm to optimize the initial 3D point-clouds and camera parameters which are produced by the SFM system. In the proposed RBA algorithm, we firstly use the Huber loss function to potentially down-weight outliers. Secondly, we split a large-scale bundle adjustment problem into some small ones by making use of the sparsity between 3D points and the cameras for reducing the requirements of memory. Thirdly, according to the inherent property of the matrix after it spare decompose, we use a fast matrix factorization algorithm to solve the normal equation to avoid calculating the inverse of large-scale matrix. Finally, we evaluate the proposed RBA method and compare it with the state-of-the-art methods on the synthetic dataset, BAL benchmark and real image datasets, respectively. Experimental results show that the proposed RBA method clearly outperforms the state-of-the-art methods on both computational cost and precision.
Similar content being viewed by others
References
Agarwal S, Snavely N, Sinmon I. (2009) Building rome in a day. In: Proceedings of the 2009 12th IEEE International Conference on Computer Vision (ICCV). pp 72–79
Agarwal S, Snavely N, Seitz S, Szeliski R (2010) Bundle adjustment in the large, In: Proceedings of the 11th European conference on computer vision (ECCV). pp. 29–42
Albl Č, Pajdla T (2014) Global camera parameterization for bundle adjustment. In: Poceedings of the 2014 International Conference on Computer Vision Theory and Applications (VISAPP). pp. 555–561
Alcantarilla P, Bartoli A, Davison A (2012) KAZE features.In: Proceedings of the 12th European conference on Computer Vision (ECCV). pp. 214–227
Bao S, Savarese S (2011) Semantic structure from motion. In: Proceedings of the 2011 I.E. Conference on Computer Vision and Pattern Recognition (CVPR). pp. 2025–2032
Blanco J, González-Jiménez J, Fernández-Madrigal J (2013) Sparser relative bundle adjustment (srba): constant-time maintenance and local optimization of arbitrarily large maps. In: Proceedings of the 2013 I.E. International Conference on Robotics and Automation (ICRA). pp. 70–77
Boyd S, Vandenberghe L (2004) Convex optimization. Cambridge university press, Cambridge
Brunier G, Fleury J, Anthony E, Gardel A, Dussouillez P (2016) Close-range airborne structure-from-motion photogrammetry for high-resolution beach morphometric surveys: examples from an embayed rotating beach. Geomorphology 261:76–88
Byröd M, Åström K (2009) Bundle adjustment using conjugate gradients with multiscale preconditioning. In: Proceedings of the 2009 British Machine Vision Conference (BMVC). pp. 1–10
Byröd M, Åström K (2010) Conjugate gradient bundle adjustment. In: Proceedings of the 11th European conference on computer vision (ECCV). pp. 114–127
Byröd M, Josephson K, Åström K (2007) Fast optimal three view triangulation, In: Proceedings of the 8th Asian Conference on Computer Vision (ACCV). pp. 549–559
Cai S, Wang C, Yan Y, Liu Y (2013) Analysis of the pencil of conics with double complex contact and its application to camera calibration. Journal of Shanghai Jiaotong University (Science) 18(5):1–6
Carlone L, Alcantailla P, Chiu H, Kira Z, Dellaert F (2014) Mining structure fragments for smart bundle adjustment. In Proceedings of the 2014 British Machine Vision Conference (BMVC). pp. 1–12
Ceylan D, Mitra N, Zheng Y, Pauly M (2014) Coupled structure-from-motion and 3D symmetry detection for urban facades. ACM Transactions on Graphics (TOG) 33(1):57–76
Chen Y, Davis T, Hager W, Rajamanickam S(2008) Algorithm 887: CHOLMOD, supernodal sparse cholesky factorization and update/downdate. ACM transactions on mathematical software 35(3):1–14
Cheng J, Leng C, Wu J, Cui H, Lu H (2014) Fast and accurate image matching with cascade hashing for 3d reconstruction. In: Proceedings of the 2014 I.E. Conference on Computer Vision and Pattern Recognition (CVPR). pp. 1–8
Chum O, Matas J (2005) Matching with PROSAC-progressive sample consensus. In: Proceedings of the 2005 I.E. Computer Society Conference on Computer Vision and Pattern Recognition (CVPR). pp. 220–226
Crandall D, Owns A, Snavely N, Hutenlocher D (2011) Discrete-continuous optimization for large-scale structure from motion. In: Proceedings of the 2011 I.E. Conference on Computer Vision and Pattern Recognition (CVPR). pp. 3001–3008
Crocco M, Rubino C, Del Bue A (2016) Structure from Motion with Objects. In: Proceedings of the 14th European conference on computer vision (ECCV).
Dellaert F, Carlson J, Lla V, Ni K, Thorpe C (2010) Subgraph-preconditioned conjugate gradients for large scale slam. In: Proceedings of the 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). pp. 2566–2571
Dong Z, Zhang G, Jia J, Bao H (2009) Keyframe-based real-time camera tracking. In: Proceedings of the 2009 12th IEEE International Conference on Computer Vision (ICCV). pp. 1538–1545
Engels C, Stewénius H, Nistér D (2006) Bundle adjustment rules. Photogrammetric computer vision 2:124–131
Eriksson A, Isaksson M, Chin T (2015) High breakdown bundle adjustment. In: Proceedings of the 2015 I.E. Winter Conference on Applications of Computer Vision (WACV). pp. 310–317
Eriksson A, Bastian J, Chin T, Isaksson M (2016) A Consensus-Based Framework for Distributed Bundle Adjustment.In: Proceedings of the 2016 I.E. Conference on Computer Vision and Pattern Recognition (CVPR).
Eudes A, Lhuillier M (2009) Error propagations for local bundle adjustment. In: Proceedings of the 2009 I.E. Conference on Computer Vision and Pattern Recognition (CVPR). pp. 2411–2418
Frahm J, George P, Gallup D, Johnson T, Raguran R, Wu C, Jen Y, Dunn E, Clipp B Lazebnik S, Pollefeys M (2010) Building Rome on a cloudless day. In: Proceedings of the 11th European conference on computer vision (ECCV). pp. 368–381
Garrigues M, Manzanera A, Bernard T (2016) Video extruder: a semi-dense point tracker for extracting beams of trajectories in real time. J Real-Time Image Proc 11(4):785–798
Geiger A, Lenz P, Urtasun R (2012) Are we ready for autonomous driving? the kitti vision benchmark suite. In: Proceedings of the 2012 I.E. conferenc on Computer Vision and Pattern Recognition (CVPR). pp. 953–960
Gong Y, Meng D, Seibel E (2015) Bound constrained bundle adjustment for reliable 3D reconstruction. Opt Express 23(8):10771–10785
Hartley R, Sturm P (1997) Triangulation. Comp Vision Image Underst (CVIU) 68(2):146–157
Hartley R, Zisserman A (2003) Multiple view geometry in computer vision. Cambridge university press, Cambridge
Heinly J, Dunn E, Frahm J (2012) Comparative evaluation of binary features. In: Proceedings of the 2012 12th European conference on Computer Vision (ECCV). pp. 759–773
Heinly J, Schonberger J, Dunn E, Frahm J (2015) Reconstructing the World* in Six Days*(As Captured by the Yahoo 100 Million Image Dataset). In: Proceedings of the 2015 I.E. Conference on Computer Vision and Pattern Recognition (CVPR). pp. 3287–3295
Hofer M, Donoser M, Bischof H (2014) Semi-Global 3D Line Modeling for Incremental Structure-from-Motion. In Proceedings of the 2014 British Machine Vision Conference (BMVC).
Holmes S, Sibley G, Klein G, Murray D (2009) A relative frame representation for fixed-time bundle adjustment in SFM. In: Proceedings of the 2009 I.E. International Conference on Robotics and Automation (ICRA). pp. 2264–2269
Huber P (1973) Robust regression: asymptotics, conjectures and Monte Carlo. The Annals of Statistics 1(5):799–821
Im S, Ha H, Choe G, Jeon H, Joo K, So Kweon I (2015) High Quality Structure From Small Motion for Rolling Shutter Cameras. In: Proceedings of the 2015 I.E. International Conference on Computer Vision (ICCV). pp. 837–845
Indelman V, Roberts R, Dellaert F (2013) Probabilistic analysis of incremental light bundle adjustment. In: Proceedings of the 2013 I.E. Workshop on Robot Vision (WORV). pp. 221–228
Jian Y, Balcan D, Dellaert F (2011) Generalized subgraph preconditioners for large-scale bundle adjustment. In: Procedings of the 2011 I.E International Conference on Computer Vision (ICCV). pp. 295–302
Jiang J, Yilmaz A (2011) Good features to track: A view geometric approach. In: proceedings of the 2011 I.E. International Conference on Computer Vision Workshops (ICCV Workshops). pp. 72–79
Kang L, Wu L, Yang Y (2014) Robust multi-view L 2 triangulation via optimal inlier selection and 3D structure refinement. Pattern Recognition (PR) 47(9):2974–2992
Klein, G. and D. Murray (2007) Parallel tracking and mapping for small AR workspaces. In: Proceedings of the 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR). pp 1–10
Klingner B, Martin D, Roseborough J (2013) Street view motion-from-structure-from-motion. In: Proceedings of the 2013 I.E. International Conference on Computer Vision (ICCV). pp. 953–960
Konolige K, Garage W (2010) Sparse Sparse Bundle Adjustment. In: Poceedings of the 2010 British Machine Vision Conference (BMVC). pp. 1–11
Kümmerle R, Grisetti G, Strasdat H, Konolige K, Burgard W (2011) g 2 o: A general framework for graph optimization. In: Proceedings of the 2011 I.E. Conference on Robotics and Automation (ICRA). pp. 3607–3613
Larnaout D, Gay-Belile V, Bourgeois S, Dhome M (2016) Fast and automatic city-scale environment modelling using hard and/or weak constrained bundle adjustments. Machine Vision and Applications (MVA): 1–20
Lee T, Hollerer T (2008) Hybrid feature tracking and user interaction for markerless augmented reality. In: Proceedings of the 2008 I.E. Virtual Reality Conference. pp. 145–152
Levenberg K (1944) A method for the solution of certain non–linear problems in least squares. Journal of Heart & Lung Transplantation the Official Publication of the International Society for Heart Transplantation 31(4):436–438
Li W, Cosker D, Lv Z, Brown M (2016) Nonrigid optical flow ground truth for real-world scenes with time-varying shading effects. In: IEEE Robotics and Automation Letters. doi:10.1109/LRA.2016.2592513
Lourakis M, Argyros A (2005) Is Levenberg-Marquardt the most efficient optimization algorithm for implementing bundle adjustment?. In: Proceedings of the 10th IEEE International Conference on. Computer Vision (ICCV). pp. 1526–1531
Lourakis M, Argyros A (2009) SBA: a software package for generic sparse bundle adjustment. ACM Transactions on Mathematical Software (TOMS) 36(1):1–30
Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vis 60(2):91–110
Lu F, Milios E (1997) Globally consistent range scan alignment for environment mapping. Auton Robot 14(4):333–349
Lu G, Ly V, Shen H, Kolagunda A, Kambhamettu C (2013a) Improving image-based localization through increasing correct feature correspondences. In: Proceedings of the 2013 International Symposium on Visual Computing. 2013. 8033:312–321
Lu G, Ly V, Kambhamettu C (2013b) Large-scale Structure-from-Motion Reconstruction with small memory consumption. In: Proceedings of the 2013 ACM International Conference on Advances in Mobile Computing & Multimedia. pp. 500–509
Lu G, Ly V, Kambhamettu C (2014) Structure-from-motion reconstruction based on weighted hamming descriptors. In: Proceedings of the 2014 International Joint Conference on Neural Networks (IJCNN). pp. 2367–2374
Lu G, Sebe N, Xu C, Kambhamttu C (2015) Memory efficient large-scale image-based localization. Multimedia Tools and Applications 74(2):479–503
Lu Z, Guerrero P, Mitra N J, Steed A (2016a) Open3D: crowd-sourced distributed curation of city models. In: Proceedings of the ACM 21st International Conference on Web3D Technology. pp. 87–94
Lu G, Yan Y, Kolagunda A, Kambhanettu C (2016b) A fast 3D indoor-localization approach based on video queries. In: Proceedings of the 2016 I.E. International Conference on Multimedia Modeling (MMM). pp. 218–230
Lu G, Yan Y, Ren L, Saponaro P, Sebe N, Kambhamettu C (2016c) Where am i in the dark: exploring active transfer learning on the use of indoor localization based on thermal imaging. Neurocomputing 173:83–92
Lv Z, Li X, Zhang B, Zhu Y, Hu J, Feng S (2016) Managing big city information based on WebVRGIS. In:IEEE Access 4:407–415
Moisan L, Moulon P, Monasse P (2012) Automatic homographic registration of a pair of images, with a contrario elimination of outliers. Image Processing On Line (IPOL) 2:56–73
Moulon P, Monasse P (2012) Unordered feature tracking made fast and easy. In: Proceedings of the 9th European Conference on Visual Media Production (CVMP). 2012.
Moulon P, Monasse P, Marlet R (2013) Global fusion of relative motions for robust, accurate and scalable structure from motion. In: Proceedings of the 2013 I.E. International Conference on Computer Vision (ICCV). pp. 3248–3255
Mouragnon E, Lhuillier M, Dhome M, Dekeyser F, Sayd P (2006) Real time localization and 3d reconstruction. In: Proceedings of the 2006 I.E. Computer Society Conference on Computer Vision and Pattern Recognition (CVPR). pp. 363–370
Ni K, Dellaert F (2012) HyperSfM. in 3D Imaging, In: Proceedings of the 2012 2th International Conference on Modeling, Processing, Visualization and Transmission (3DIMPVT).pp. 144–151
Ni K, Steedly D, Dellaert F (2007) Out-of-core bundle adjustment for large-scale 3d reconstruction. In: Proceedings of the 11th IEEE International Conference on Computer Vision (ICCV). pp. 1–8
Nistér D (2008) An efficient solution to the five-point relative pose problem. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 26(6):756–770
Nurutdinova I, Fitzgibbon A (2015) Towards Pointless Structure from Motion: 3D reconstruction and camera parameters from general 3D curves. In: Proceedings of the 2015 I.E. International Conference on Computer Vision (ICCV). pp. 2363–2371
Poling B, Lerman G, Szlam A (2014) Better feature tracking through subspace constraints. In: Proceedings of the 2014 I.E. Conference on Computer Vision and Pattern Recognition (CVPR). pp. 3454–3461
Powell M (1970) A hybrid method for nonlinear equations. Numerical methods for nonlinear algebraic equations 7:87–114
de la Puente P, Rodríguez-Losada D (2014) Feature based graph-SLAM in structured environments. Auton Robot 37(3):243–260
Raguram R, Chum O, Pollefeys M, Matas J, Frahm J (2013) Usac: a universal framework for random sample consensus. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 35(8):2022–2038
Resch B, Lensch H, Wang O, Pollefeys M, Sokine-Hornung A (2015) Scalable structure from motion for densely sampled videos. In: Proceedings of the 2015 I.E. Conference on Computer Vision and Pattern Recognition (CVPR). pp. 3936–3944
Rohith M, Kambhamettu C (2012) Application of heterogenous motion models towards structure recovery from motion. In: Proceedings of the 2012 Asian Conference on Computer Vision (ACCV). pp. 622–635
Saponaro P, Sorensen S, Rhein S, Mahoney A, Kanbhamettu C (2014) Reconstruction of textureless regions using structure from motion and image-based interpolation. In: Proceedings of the 2014 I.E. International Conference on Image Processing (ICIP). pp. 1847–1851
Schönberger J, Frahm J (2016) Structure-from-motion revisited. In: Proceedings of the 2011 I.E. Conference on Computer Vision and Pattern Recognition (CVPR).
Shah R, Deshpande A, Narayanan P (2015) Multistage SFM: A Coarse-to-Fine Approach for 3D Reconstruction. arXiv preprint arXiv:1512.06235
Shum H, Ke Q, Zhang Z (1999) Efficient bundle adjustment with virtual key frames: A hierarchical approach to multi-frame structure from motion. In: Proceedings of the 1999 I.E. Computer Society Conference on Computer Vision and Pattern Recognition (CVPR). pp. 538–543
Snavely N, Seitz SM, Szeliski R (2006) Photo tourism: exploring photo collections in 3D. ACM transactions on graphics (TOG) 25(3):835–846
Sorensen S, Saponaro P, Rhein S, Kambhamettu C (2015a) Multimodal Stereo Vision For Reconstruction In The Presence Of Reflection. In: Proceedings of the 2015 British machine vision conference (BMVC).
Sorensen S,Kolagunda A, Saponaro P (2015b) Refractive stereo ray tracing for reconstructing underwater structures. In: Proceedings of the 2015 I.E. International Conference on Image Processing (ICIP). pp. 1712–1716
Steedly D, Essa I (2001) Propagation of innovative information in non-linear least-squares structure from motion. In: Proceedings of the 2001 8th IEEE International Conference on Computer Vision (ICCV). pp. 223–229
Steedly D, Essa I, Dellaert F (2003) Spectral partitioning for structure from motion. In: Proceedings of the 2003 9th IEEE International Conference on Computer Vision (ICCV). pp. 996–1003
Su T, Cao Z, Lv Z, Liu C, Li X (2016) Multi-dimensional visualization of large-scale marine hydrological environmental data. Adv Eng Softw 95:7–15
Zhang X, Han Y, Hao D, Lv Z (2016) ARGIS-based Outdoor Underground Pipeline Information System Journal of Visual Communication and Image Representation (IVCIP), 2016
Tola E, Strecha C, Fua P (2011) Efficient large-scale multi-view stereo for ultra high-resolution image sets. Machine Vision & Applications 23(5):903–920
Ylimäki M, Kannala J, Holappa J, Brandt S (2015) Fast and accurate multi-view reconstruction by multi-stage prioritised matching. IET Comput Vis 9(4):576–587
Yan Y, Ricci E, Subranmanian R, Liu G, Lanz O, Sebe N (2016) A multi-task learning framework for head pose estimation under target motion. IEEE Trans Pattern Anal Mach Intell 38(6):1070–1083
Xia Y, Zhang L, Hong R, Nie L, Yan Y, Ling S (2016) Perceptually guided photo retargeting. In: IEEE Transactions on Cybernetics 47(3):1–13
Xiao J, Owens A, Torralba A. (2013) SUN3D: A database of big spaces reconstructed using sfm and object labels. In: Proceedings of the 2013 I.E. International Conference on Computer Vision (ICCV). pp. 1625–1632
Zheng E, Wu C (2015) Structure from Motion Using Structure-less Resection. In: Proceedings of the 2015 I.E. International Conference on Computer Vision (ICCV). pp. 2075–2083
Triggs B, McLuchlan P, Hartley R, Fitzgibbon A (2000) Bundle adjustment—a modern synthesis. In: Lecture Notes in computer Science 1883(1883):298–372
Vo M, Narasimhan S,Sheikh Y (2016) Spatiotemporal Bundle Adjustment for Dynamic 3D Reconstruction. In : Proceedings of the 2016 I.E. Conference in Computer Vision and Pattern Recognition (CVPR). pp. 1–9
Wu C, Agarwal S, Curless B, Seitz S (2011) Multicore bundle adjustment. In: Proceedings of the 2011 I.E. Conference on Computer Vision and Pattern Recognition (CVPR). pp. 3057–3064
Zhao L, Huang S, Sun Y, Yan L, Dissanayake G (2015a) ParallaxBA: bundle adjustment using parallax angle feature parametrization. The International Journal of Robotics Research (IJRR) 34(4–5):493–516
Thomee B, Shamma D, Friedland G, Elizalde B, Ni K, Poland D, Borth D, Li L (2016) YFCC100M: the new data in multimedia research. Commun ACM 59(2):64–73
Zach C (2014) Robust bundle adjustment revisited, In: Proceedings of the 13th European conference on computer vision (ECCV). pp. 772–787
Wilson K, Snavely N (2014) Robust global translations with 1dsfm. In: Proceedings of the 13th European conference on computer vision (ECCV). pp. 61–75
Zheng M, Zhang Y (2016) DEM-aided bundle adjustment with multisource satellite imagery: ZY-3 and GF-1 in large areas. IEEE Geosci Remote Sens Lett 13(6):880–884
Zhang G, Vela P (2015) Good Features to Track for Visual SLAM. In: Proceedings of the 2015 I.E. Conference on Computer Vision and Pattern Recognition (CVPR). pp. 1373–1382
Zhao L, Li X, Xiao J, Wu F, Zhuang Y (2015b) Metric learning driven multi-task structured output optimization for robust keypoint tracking. In Proceedings of the 29th {AAAI} Conference on Artificial Intelligence. pp. 3864–3870
Svärm L, Simayijiang Z, Enqvist O, Olsson C (2012) Point track creation in unordered image collections using Gomory-Hu trees. In: Poceedings of the 2012 21st International Conference on Pattern Recognition (ICPR). pp. 2116–2119
Xi Y, Xia J, Chan R (2014) A fast randomized eigensolver with structured LDL factorization update. SIAM Journal on Matrix Analysis and Applications 35(3):974–996
Zach C (2010) ETH-V3D Structure-and-Motion software.© 2010-2011. ETH Zurich, 2010.
Wu C (2011) SiftGPU: A GPU implementation of scale invariant feature transform (SIFT)(2007). URL http://cs. unc.edu/~ ccwu/siftgpu, 2011.
Zhang G, Liu H, Dong Z, Jia J, Wong, T, Bao H (2015) ENFT: Efficient Non-Consecutive Feature Tracking for Robust Structure-from-Motion. arXiv preprint arXiv:1510.08012
Wu C (2013) Towards linear-time incremental structure from motion. In: Proceedings of the 2013 International Conference on 3D Vision (3DV). pp. 127–134
Sweeney C, Sattler T, Hollerer T, Turk M, Pollefey M (2015) Optimizing the Viewing Graph for Structure-from-Motion. In: Proceedings of the 2015 I.E. International Conference on Computer Vision (ICCV). pp. 801–809
Wang T, Kohli P, Mitra N (2015) Dynamic SFM: detecting scene changes from image pairs. Computer Graphics Forum 34(5):177–189
Acknowledgements
This work is partly supported by the grants of the National Science Foundation of China, Nos. 61370167, 61673157, 61305093, and 61402018, the grant of the Natural Science Foundation of Anhui Province, Nos. KJ2014ZD27 and JZ2015AKZR0664, and also supported by the National Key Research and Development Plan under Grant No. 2016YFC0800100. The authors would like to thank anonymous reviewers for their helpful and constructive comments that greatly improve the paper.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Cao, M., Li, S., Jia, W. et al. Robust bundle adjustment for large-scale structure from motion. Multimed Tools Appl 76, 21843–21867 (2017). https://doi.org/10.1007/s11042-017-4581-5
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-017-4581-5