Ground Truth Evaluation of Stereo Algorithms for Real World Applications

  • Sandino Morales
  • Reinhard Klette
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6469)


Current stereo algorithms are capable to calculate accurate (as defined, e.g., by needs in vision-based driver assistance) dense disparity maps in real time. They have become the source of three-dimensional data for several indoor and outdoor applications. However, ground truth-based evaluation of such algorithms has been typically limited to data sets generated indoors in laboratories. In this paper we present a new approach to evaluate stereo algorithms using ground-truth over real world data sets. Ground truth is generated using range measurements acquired with a high-end laser range-finder. For evaluating as many points as possible in a given disparity map, we use two evaluation approaches: A direct comparison for those pixels with available range data, and a confidence measure for the remaining pixels.


Performance evaluation stereo algorithms laser range finder 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Banks, J., Corke, P.: Quantitative evaluation of matching methods and validity measures for stereo vision. Int. J. Robotics Research 20, 512–532 (2001)CrossRefGoogle Scholar
  2. 2.
    Eid, A., Farag, A.: A unified framework for performance evaluation of 3-d reconstruction techniques. In: Proc. Comp. Vision Pattern Recognition Workshop, vol. 3, pp. 33–41 (2004)Google Scholar
  3. 3.
    Egnal, G., Mintz, M., Wildes, R.: A stereo confidence metric using single view imagery with comparison to five alternative approaches. Image Vision Computing 22, 943–957 (2004)CrossRefGoogle Scholar
  4. 4.
    .enpeda. Group, University of Auckland: EISATS (.enpeda. sequence analysis test site) (2010),
  5. 5.
    Felzenszwalb, P.F., Huttenlocher, D.P.: Efficient belief propagation for early vision. Int. J. Comp. Vision 70, 41–54 (2006)CrossRefGoogle Scholar
  6. 6.
    Gherardi, R.: Confidence-based cost modulation for stereo matching. In: Proc. ICPR, pp. 1–4 (2008)Google Scholar
  7. 7.
    Guan, S., Klette, R., Woo, Y.W.: Belief propagation for stereo analysis of night-vision sequences. In: PSIVT 2009. LNCS, vol. 5414, pp. 932–943 (2009)Google Scholar
  8. 8.
    Haeusler, R., Klette, R.: Benchmarking stereo data (Not the matching algorithms). In: Goesele, M., Roth, S., Kuijper, A., Schiele, B., Schindler, K. (eds.) Pattern Recognition. LNCS, vol. 6376, pp. 383–392. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  9. 9.
    Hirschmüller, H.: Accurate and efficient stereo processing by semi-global matching and mutual information. In: Proc. CVPR, pp. 807–814 (2005)Google Scholar
  10. 10.
    Huang, F., Klette, R., Scheibe, K.: Panoramic Imaging: Sensor-Line Cameras and Laser Range-Finders. Wiley, Chichester (2008)CrossRefGoogle Scholar
  11. 11.
    Klette, R., Vaudrey, T., Wiest, J., Haeusler, R., Jiang, R., Morales, S.: Current challenges in vision-based driver assistance. In: Progress in Combinat, pp. 3–32. Image Analysis, Research Publ. Services, Singapore (2010)Google Scholar
  12. 12.
    Kolmogorov, V., Zabih, R.: Multi-camera scene reconstruction via graph cuts. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002. LNCS, vol. 2352, pp. 82–96. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  13. 13.
    Lepetit, V., Moreno-Noguer, F., Fua, P.: Accurate non-iterative o(n)solution to the PNP problem. In: Proc. ICCV, pp. 874–885 (2007)Google Scholar
  14. 14.
    Liu, Z., Klette, R.: Approximate ground truth for stereo and motion analysis on real-world sequences. In: PSIVT 2009. LNCS, vol. 5414, pp. 874–885 (2009)Google Scholar
  15. 15.
    Morales, S., Klette, R.: A third eye for performance evaluation in stereo sequence analysis. In: Jiang, X., Petkov, N. (eds.) CAIP 2009. LNCS, vol. 5702, pp. 1078–1086. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  16. 16.
    Mordohai, P.: The self-aware matching measure for stereo. In: Proc. ICCV, pp. 1841–1848 (2009)Google Scholar
  17. 17.
    Murray, D., Little, J.J.: Using real-time stereo vision for mobile robot navigation. Aut. Robots 8, 161–171 (2000)CrossRefGoogle Scholar
  18. 18.
    Ohta, Y., Kanade, T.: Stereo by two-level dynamic programming. In: Proc. Int. Joint Conf. Artificial Int., pp. 1120–1126 (1985)Google Scholar
  19. 19.
    Reulke, R., Luber, A., Haberjahn, M., Piltz, B.: Validierung von mobilen Stereokamerasystemen in einem 3D-Testfeld. In: Proc. 3D-NordOst (2009)Google Scholar
  20. 20.
    Satoh, Y., Sakaue, K.: An omnidirectional stereo vision-based smart wheelchair. J. Image Video Processing 2007, 1–11 (2007)CrossRefGoogle Scholar
  21. 21.
    Seelinger, M., Yoder, J.D.: Automatic pallet engagement by a vision guided forklift. In: Proc. IEEE Int. Conf. Robotics Automation, pp. 4068–4073 (2005)Google Scholar
  22. 22.
    Steingrube, P., Gehrig, S., Franke, U.: Performance evaluation of stereo algorithms for automotive applications. In: Proc. Int. Conf. Comp. Vision Systems, pp. 285–394 (2009)Google Scholar
  23. 23.
    Szeliski, R.: Prediction error as a quality metric for motion and stereo. In: Proc. ICCV, pp. 781–788 (1999)Google Scholar
  24. 24.
    Velodyne Lidar Inc.: Velodyne’s HDL-64E S2 user manual,

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Sandino Morales
    • 1
  • Reinhard Klette
    • 1
  1. 1.enpeda.. group, Dept. Computer ScienceUniversity of AucklandNew Zealand

Personalised recommendations