Advertisement

Semi-dense 3D Reconstruction with a Stereo Event Camera

  • Yi ZhouEmail author
  • Guillermo Gallego
  • Henri Rebecq
  • Laurent Kneip
  • Hongdong Li
  • Davide Scaramuzza
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11205)

Abstract

Event cameras are bio-inspired sensors that offer several advantages, such as low latency, high-speed and high dynamic range, to tackle challenging scenarios in computer vision. This paper presents a solution to the problem of 3D reconstruction from data captured by a stereo event-camera rig moving in a static scene, such as in the context of stereo Simultaneous Localization and Mapping. The proposed method consists of the optimization of an energy function designed to exploit small-baseline spatio-temporal consistency of events triggered across both stereo image planes. To improve the density of the reconstruction and to reduce the uncertainty of the estimation, a probabilistic depth-fusion strategy is also developed. The resulting method has no special requirements on either the motion of the stereo event-camera rig or on prior knowledge about the scene. Experiments demonstrate our method can deal with both texture-rich scenes as well as sparse scenes, outperforming state-of-the-art stereo methods based on event data image representations.

Notes

Acknowledgment

The research leading to these results is supported by the Australian Centre for Robotic Vision and the National Center of Competence in Research (NCCR) Robotics, through the Swiss National Science Foundation, the SNSF-ERC Starting Grant and the NCCR Ph.D. Exchange Scholarship Programme. Yi Zhou also acknowledges the financial support from the China Scholarship Council for his Ph.D. Scholarship No. 201406020098.

Supplementary material

Supplementary material 1 (mp4 84209 KB)

474172_1_En_15_MOESM2_ESM.pdf (189 kb)
Supplementary material 2 (pdf 188 KB)

References

  1. 1.
    Lichtsteiner, P., Posch, C., Delbruck, T.: A 128 \(\times \) 128 120 dB 15 \(\upmu \)s latency asynchronous temporal contrast vision sensor. IEEE J. Solid-State Circuits 43(2), 566–576 (2008).  https://doi.org/10.1109/JSSC.2007.914337CrossRefGoogle Scholar
  2. 2.
    Mueggler, E., Huber, B., Scaramuzza, D.: Event-based, 6-DOF pose tracking for high-speed maneuvers. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2761–2768 (2014).https://doi.org/10.1109/IROS.2014.6942940
  3. 3.
    Lagorce, X., Meyer, C., Ieng, S.H., Filliat, D., Benosman, R.: Asynchronous event-based multikernel algorithm for high-speed visual features tracking. IEEE Trans. Neural Netw. Learn. Syst. 26(8), 1710–1720 (2015).  https://doi.org/10.1109/TNNLS.2014.2352401MathSciNetCrossRefGoogle Scholar
  4. 4.
    Zhu, A.Z., Atanasov, N., Daniilidis, K.: Event-based feature tracking with probabilistic data association. In: IEEE International Conference on Robotics and Automation (ICRA), pp. 4465–4470 (2017). https://doi.org/10.1109/ICRA.2017.7989517
  5. 5.
    Gallego, G., Lund, J.E.A., Mueggler, E., Rebecq, H., Delbruck, T., Scaramuzza, D.: Event-based, 6-DOF camera tracking from photometric depth maps. IEEE Trans. Pattern Anal. Mach. Intell. 40(10), 2402–2412 (2018).  https://doi.org/10.1109/TPAMI.2017.2658577CrossRefGoogle Scholar
  6. 6.
    Gallego, G., Scaramuzza, D.: Accurate angular velocity estimation with an event camera. IEEE Robot. Autom. Lett. 2, 632–639 (2017).  https://doi.org/10.1109/LRA.2016.2647639CrossRefGoogle Scholar
  7. 7.
    Mueggler, E., Gallego, G., Rebecq, H., Scaramuzza, D.: Continuous-time visual-inertial odometry with event cameras. IEEE Trans. Robot. (2018, to appear). https://doi.org/10.1109/TRO.2018.2858287
  8. 8.
    Gehrig, D., Rebecq, H., Gallego, G., Scaramuzza, D.: Asynchronous, photometric feature tracking using events and frames. In: European Conference on Computer Vision (ECCV) (2018, to appear).  https://doi.org/10.1007/978-3-030-01258-8_46
  9. 9.
    Conradt, J., Cook, M., Berner, R., Lichtsteiner, P., Douglas, R.J., Delbruck, T.: A pencil balancing robot using a pair of AER dynamic vision sensors. In: IEEE International Symposium on Circuits and Systems (ISCAS), pp. 781–784 (2009). https://doi.org/10.1109/ISCAS.2009.5117867
  10. 10.
    Delbruck, T., Lang, M.: Robotic goalie with 3 ms reaction time at 4% CPU load using event-based dynamic vision sensor. Front. Neurosci. 7, 223 (2013).  https://doi.org/10.3389/fnins.2013.00223CrossRefGoogle Scholar
  11. 11.
    Rebecq, H., Gallego, G., Scaramuzza, D.: EMVS: event-based multi-view stereo. In: British Machine Vision Conference (BMVC), September 2016. https://doi.org/10.5244/C.30.63
  12. 12.
    Kim, H., Leutenegger, S., Davison, A.J.: Real-time 3D reconstruction and 6-DOF tracking with an event camera. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9910, pp. 349–364. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46466-4_21CrossRefGoogle Scholar
  13. 13.
    Rebecq, H., Gallego, G., Mueggler, E., Scaramuzza, D.: EMVS: event-based multi-view stereo–3D reconstruction with an event camera in real-time. Int. J. Comput. Vis. 1–21 (2017). https://doi.org/10.1007/s11263-017-1050-6
  14. 14.
    Rebecq, H., Horstschäfer, T., Gallego, G., Scaramuzza, D.: EVO: a geometric approach to event-based 6-DOF parallel tracking and mapping in real-time. IEEE Robot. Autom. Lett. 2, 593–600 (2017).  https://doi.org/10.1109/LRA.2016.2645143CrossRefGoogle Scholar
  15. 15.
    Rebecq, H., Horstschaefer, T., Scaramuzza, D.: Real-time visual-inertial odometry for event cameras using keyframe-based nonlinear optimization. In: British Machine Vision Conference (BMVC), September 2017Google Scholar
  16. 16.
    Rosinol Vidal, A., Rebecq, H., Horstschaefer, T., Scaramuzza, D.: Ultimate SLAM? Combining events, images, and IMU for robust visual SLAM in HDR and high speed scenarios. IEEE Robot. Autom. Lett. 3(2), 994–1001 (2018).  https://doi.org/10.1109/LRA.2018.2793357CrossRefGoogle Scholar
  17. 17.
    Censi, A., Scaramuzza, D.: Low-latency event-based visual odometry. In: IEEE International Conference on Robotics and Automation (ICRA), pp. 703–710 (2014). https://doi.org/10.1109/ICRA.2014.6906931
  18. 18.
    Kueng, B., Mueggler, E., Gallego, G., Scaramuzza, D.: Low-latency visual odometry using event-based feature tracks. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Daejeon, pp. 16–23, October 2016. https://doi.org/10.1109/IROS.2016.7758089
  19. 19.
    Weikersdorfer, D., Adrian, D.B., Cremers, D., Conradt, J.: Event-based 3D SLAM with a depth-augmented dynamic vision sensor. In: IEEE International Conference on Robotics and Automation (ICRA), pp. 359–364, June 2014. https://doi.org/10.1109/ICRA.2014.6906882
  20. 20.
    Kogler, J., Humenberger, M., Sulzbachner, C.: Event-based stereo matching approaches for frameless address event stereo data. In: Bebis, G. (ed.) ISVC 2011. LNCS, vol. 6938, pp. 674–685. Springer, Heidelberg (2011).  https://doi.org/10.1007/978-3-642-24028-7_62CrossRefGoogle Scholar
  21. 21.
    Rogister, P., Benosman, R., Ieng, S.H., Lichtsteiner, P., Delbruck, T.: Asynchronous event-based binocular stereo matching. IEEE Trans. Neural Netw. Learn. Syst. 23(2), 347–353 (2012).  https://doi.org/10.1109/TNNLS.2011.2180025CrossRefGoogle Scholar
  22. 22.
    Camunas-Mesa, L.A., Serrano-Gotarredona, T., Ieng, S.H., Benosman, R.B., Linares-Barranco, B.: On the use of orientation filters for 3D reconstruction in event-driven stereo vision. Front. Neurosci. 8, 48 (2014).  https://doi.org/10.3389/fnins.2014.00048CrossRefGoogle Scholar
  23. 23.
    Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision, 2nd edn. Cambridge University Press, New York (2003).  https://doi.org/10.1017/CBO9780511811685CrossRefzbMATHGoogle Scholar
  24. 24.
    Piatkowska, E., Belbachir, A.N., Gelautz, M.: Asynchronous stereo vision for event-driven dynamic stereo sensor using an adaptive cooperative approach. In: International Conference on Computer Vision Workshops (ICCVW), pp. 45–50 (2013). https://doi.org/10.1109/ICCVW.2013.13
  25. 25.
    Osswald, M., Ieng, S.H., Benosman, R., Indiveri, G.: A spiking neural network model of 3D perception for event-based neuromorphic stereo vision systems. Sci. Rep. 7, 1–12 (2017).  https://doi.org/10.1038/srep40703. 40703CrossRefGoogle Scholar
  26. 26.
    Marr, D., Poggio, T.: Cooperative computation of stereo disparity. Science 194(4262), 283–287 (1976).  https://doi.org/10.1126/science.968482CrossRefGoogle Scholar
  27. 27.
    Schraml, S., Belbachir, A.N., Bischof, H.: An event-driven stereo system for real-time 3-D 360 panoramic vision. IEEE Trans. Ind. Electron. 63(1), 418–428 (2016).  https://doi.org/10.1109/tie.2015.2477265CrossRefGoogle Scholar
  28. 28.
    Schraml, S., Belbachir, A.N., Bischof, H.: Event-driven stereo matching for real-time 3D panoramic vision. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 466–474, June 2015. https://doi.org/10.1109/CVPR.2015.7298644
  29. 29.
    Gallego, G., Rebecq, H., Scaramuzza, D.: A unifying contrast maximization framework for event cameras, with applications to motion, depth, and optical flow estimation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3867–3876 (2018). https://doi.org/10.1109/CVPR.2018.00407
  30. 30.
    Newcombe, R.A., Lovegrove, S.J., Davison, A.J.: DTAM: dense tracking and mapping in real-time. In: International Conference on Computer Vision (ICCV), pp. 2320–2327, November 2011. https://doi.org/10.1109/ICCV.2011.6126513
  31. 31.
    Benosman, R., Ieng, S.H., Rogister, P., Posch, C.: Asynchronous event-based Hebbian epipolar geometry. IEEE Trans. Neural Netw. 22(11), 1723–1734 (2011).  https://doi.org/10.1109/TNN.2011.2167239CrossRefGoogle Scholar
  32. 32.
    Lagorce, X., Orchard, G., Gallupi, F., Shi, B.E., Benosman, R.: HOTS: a hierarchy of event-based time-surfaces for pattern recognition. IEEE Trans. Pattern Anal. Mach. Intell. PP(99), 1 (2016).  https://doi.org/10.1109/TPAMI.2016.2574707CrossRefGoogle Scholar
  33. 33.
    Liu, S.C., Delbruck, T.: Neuromorphic sensory systems. Curr. Opin. Neurobiol. 20(3), 288–295 (2010).  https://doi.org/10.1016/j.conb.2010.03.007CrossRefGoogle Scholar
  34. 34.
    Zhu, A.Z., Thakur, D., Ozaslan, T., Pfrommer, B., Kumar, V., Daniilidis, K.: The multivehicle stereo event camera dataset: an event camera dataset for 3D perception. IEEE Robot. Autom. Lett. 3(3), 2032–2039 (2018).  https://doi.org/10.1109/lra.2018.2800793CrossRefGoogle Scholar
  35. 35.
    Mur-Artal, R., Tardos, J.D.: Probabilistic semi-dense mapping from highly accurate feature-based monocular SLAM. In: Robotics: Science and Systems (RSS) (2015). https://doi.org/10.15607/RSS.2015.XI.041
  36. 36.
    Mueggler, E., Rebecq, H., Gallego, G., Delbruck, T., Scaramuzza, D.: The event-camera dataset and simulator: event-based data for pose estimation, visual odometry, and SLAM. Int. J. Robot. Res. 36, 142–149 (2017).  https://doi.org/10.1177/0278364917691115CrossRefGoogle Scholar
  37. 37.
    Brandli, C., Berner, R., Yang, M., Liu, S.C., Delbruck, T.: A 240 \(\times \) 180 130 dB 3 us latency global shutter spatiotemporal vision sensor. IEEE J. Solid-State Circuits 49(10), 2333–2341 (2014).  https://doi.org/10.1109/JSSC.2014.2342715CrossRefGoogle Scholar
  38. 38.
    Furgale, P., Rehder, J., Siegwart, R.: Unified temporal and spatial calibration for multi-sensor systems. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (2013). https://doi.org/10.1109/IROS.2013.6696514
  39. 39.
    Hosni, A., Rhemann, C., Bleyer, M., Rother, C., Gelautz, M.: Fast cost-volume filtering for visual correspondence and beyond. IEEE Trans. Pattern Anal. Mach. Intell. 35(2), 504–511 (2013).  https://doi.org/10.1109/tpami.2012.156CrossRefGoogle Scholar
  40. 40.
    Hirschmuller, H.: Stereo processing by semiglobal matching and mutual information. IEEE Trans. Pattern Anal. Mach. Intell. 30(2), 328–341 (2008).  https://doi.org/10.1109/tpami.2007.1166CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Yi Zhou
    • 1
    • 2
    Email author
  • Guillermo Gallego
    • 3
  • Henri Rebecq
    • 3
  • Laurent Kneip
    • 4
  • Hongdong Li
    • 1
    • 2
  • Davide Scaramuzza
    • 3
  1. 1.Australian National UniversityCanberraAustralia
  2. 2.Australian Centre for Robotic VisionBrisbaneAustralia
  3. 3.Departments of Informatics and NeuroinformaticsUniversity of Zurich and ETH ZurichZurichSwitzerland
  4. 4.School of Information Science and TechnologyShanghaiTech UniversityShanghaiChina

Personalised recommendations