Abstract
Evaluation of stereo-analysis algorithms is usually done by analysing the performance of stereo matchers on data sets with available ground truth. The trade-off between precise results, obtained with this sort of evaluation, and the limited amount (in both, quantity and diversity) of data sets, needs to be considered if the algorithms are required to analyse real-world environments. This chapter discusses a technique to objectively evaluate the performance of stereo-analysis algorithms using real-world image sequences. The lack of ground truth is tackled by incorporating an extra camera into a multi-view stereo camera system. The relatively simple hardware set-up of the proposed technique can easily be reproduced for specific applications.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Badino, H., Franke, U., Mester, R.: Free space computation using stochastic occupancy grids and dynamic programming. In: Proc. Dynamic Vision, ICCV Workshop, pp. 1–12 (2007)
Banks, J., Corke, P.: Quantitative evaluation of matching methods and validity measures for stereo vision. Int. J. Robotics Research 20, 512–532 (2001)
Bolles, R., Baker, H., Hannah, M.: The JISCT stereo evaluation. In: ARPA Image Understanding Workshop, pp. 263–274 (1993)
CMU/VASC. Stereo image data base, http://vasc.ri.cmu.edu/idb/html/stereo/ (retrieved 2012)
Computer Vision Group, University of Bonn. Stereo images with ground truth disparity and occlusion, http://www.uni-bonn.de/~uzs75l/MRTStereo/stereo_data/index.html (retrieved 2012)
DAGM 2011, adverse vision condition challenge, http://www.dagm2011.org/adverse-vision-conditions-challenge.html (retrieved 2012)
The .enpeda.. project, The University of Auckland. EISATS, Set 2, http://www.mi.auckland.ac.nz/EISATS (retrieved 2012)
Faugeras, O., Fua, P., Hotz, B., Ma, R., Robert, L., Thonnat, M., Zhang, Z.: Quantitative and qualitative comparison of some area and feature-based stereo-analysis algorithms. In: Proc. Workshop Robust Computer Vision, pp. 1–26 (1992)
Felzenszwalb, P., Huttenlocher, D.: Efficient belief propagation for early vision. Int. J. Computer Vision 70, 41–54 (2006)
Georgescu, B., Meer, P.: Point matching under large image deformations and illumination changes. IEEE Trans. Pattern Anal. Mach. Intel. 26, 674–688 (2004)
Gherardi, R.: Confidence-based cost modulation for stereo matching. In: Proc. ICPR (2008) 978-1-4244-2175-6
Gülch, E.: Results of test on image matching of ISPRS WG III/4. ISPRS J. Photogrammetry Remote Sensing 46, 1–18 (1991)
Haeusler, R., Klette, R.: Benchmarking Stereo Data (Not the Matching Algorithms). In: Goesele, M., Roth, S., Kuijper, A., Schiele, B., Schindler, K. (eds.) DAGM 2010. LNCS, vol. 6376, pp. 383–392. Springer, Heidelberg (2010)
Hermann, S., Vaudrey, T.: The gradient - a powerful and robust cost function for stereo matching. In: Proc. IVCNZ (2010) 978-1-4244-9631-0
Hermann, S., Morales, S., Klette, R.: Half-resolution semi-global stereo matching. In: Proc. IEEE Symp. IV, pp. 201–206 (2011)
Hermann, S., Klette, R.: Evaluation of a New Coarse-to-Fine Strategy for Fast Semi-Global Stereo Matching. In: Ho, Y.-S. (ed.) PSIVT 2011, Part I. LNCS, vol. 7087, pp. 395–406. Springer, Heidelberg (2011)
Hirschmüller, H.: Accurate and efficient stereo processing by semi-global matching and mutual information. In: Proc. CVPR, vol. 2, pp. 807–814 (2005)
JISCT Stereo Images, http://vasc.ri.cmu.edu/idb/html/jisct/index.html (retrieved 2012)
Klappstein, J., Vaudrey, T., Rabe, C., Wedel, A., Klette, R.: Moving Object Segmentation using Optical Flow and Depth Information. In: Wada, T., Huang, F., Lin, S. (eds.) PSIVT 2009. LNCS, vol. 5414, pp. 611–623. Springer, Heidelberg (2009)
Klaus, A., Sormann, M., Karner, K.: Segment-based stereo matching using belief propagation and a self-adapting dissimilarity measure. In: Proc. ICPR (2006), 10.1109/ICPR.2006.1033
Klette, R., Schlüns, K., Koschan, A.: Computer vision: three-dimensional data from images. Springer, Singapore (1998)
Klette, R., Zamperoni, P.: Handbook of Image Processing Operators. John Wiley & Sons, Inc. (1996)
Klette, R., Krüger, N., Vaudrey, T., Pauwels, K., Hulle, M., Morales, S., Kandil, F., Haeusler, R., Pugeault, N., Rabe, C., Leppe, M.: Performance of correspondence algorithms in vision-based driver assistance using an online image sequence database. IEEE Trans. Vehicular Technology 60, 2012–2026 (2011)
Kogler, J., Hemetsberger, H., Alefs, B., Kubinger, W., Travis, W.: Embedded stereo vision system for intelligent autonomous vehicles. In: Proc. IEEE Symp. IV, pp. 64–69 (2006)
Kolmogorov, V., Zabih, R.: Graph cut algorithms for binocular stereo with occlusions. In: Math. Models in Computer Vision: The Handbook, pp. 423–437. Springer (2005)
Kondermann, D., Meister, S., Lauer, P.: An outdoor stereo camera system for the generation of real-world benchmark datasets with ground truth. Universität Heidelberg HCI, Technical Rep. (2011)
Leclerc, Y.G., Luong, Q.-T., Fua, P.: Measuring the Self-Consistency of Stereo Algorithms. In: Vernon, D. (ed.) ECCV 2000. LNCS, vol. 1842, pp. 282–298. Springer, Heidelberg (2000)
Liu, Z., Klette, R.: Approximate Ground Truth for Stereo and Motion Analysis on Real-World Sequences. In: Wada, T., Huang, F., Lin, S. (eds.) PSIVT 2009. LNCS, vol. 5414, pp. 874–885. Springer, Heidelberg (2009)
Mohan, R., Medioni, G., Nevatia, R.: Stereo error detection, correction and evaluation. IEEE Trans. Pattern Analysis Machine Intelligence 11, 113–120 (1989)
Morales, S., Klette, R.: A Third Eye for Performance Evaluation in Stereo Sequence Analysis. In: Jiang, X., Petkov, N. (eds.) CAIP 2009. LNCS, vol. 5702, pp. 1078–1086. Springer, Heidelberg (2009)
Morales, S., Klette, R.: Ground Truth Evaluation of Stereo Algorithms for Real World Applications. In: Koch, R., Huang, F. (eds.) ACCV 2010 Workshops, Part II. LNCS, vol. 6469, pp. 152–162. Springer, Heidelberg (2011)
Mordohai, P.: The self-aware matching measure for stereo. In: Proc. ICCV, pp. 1841–1848 (2009)
Satoh, Y., Sakaue, K.: An omnidirectional stereo vision-based smart wheelchair. EURASIP J. Image Video Processing, 1–12 (2007)
Scharstein, D., Szeliski, R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. J. Computer Vision 47, 7–42 (2001)
Schauwecker, K., Morales, S., Hermann, S., Klette, R.: A comparative study of stereo-matching algorithms for road-modelling in the presence of windscreen wipers. In: Proc. IEEE Symp. IV, pp. 7–12 (2011)
Steingrube, P., Gehrig, S.K., Franke, U.: Performance Evaluation of Stereo Algorithms for Automotive Applications. In: Fritz, M., Schiele, B., Piater, J.H. (eds.) ICVS 2009. LNCS, vol. 5815, pp. 285–294. Springer, Heidelberg (2009)
Szeliski, R.: Prediction error as a quality metric for motion and stereo. In: Proc. ICCV, pp. 781–788 (1999)
van der Mark, W., Gavrila, M.: Real-time dense stereo for intelligent vehicles. IEEE Trans. Intelligent Transportation Systems 7, 38–50 (2006)
Vaudrey, T., Rabe, C., Klette, R., Milburn, J.: Differences between stereo and motion behaviour on synthetic and real-world stereo sequences. In: Proc. IVCNZ, pp. 1–6 (2008)
Vaudrey, T., Morales, S., Wedel, A., Klette, R.: Generalized residual images effect on illumination artifact removal for correspondence algorithms. Pattern Recognition 44, 2034–2046 (2011)
Zabih, R., Woodfill, J.: Non-Parametric Local Transforms for Computing Visual Correspondence. In: Eklundh, J.-O. (ed.) ECCV 1994, Part II. LNCS, vol. 801, pp. 151–158. Springer, Heidelberg (1994)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Morales, S., Hermann, S., Klette, R. (2012). Real-World Stereo-Analysis Evaluation. In: Dellaert, F., Frahm, JM., Pollefeys, M., Leal-Taixé, L., Rosenhahn, B. (eds) Outdoor and Large-Scale Real-World Scene Analysis. Lecture Notes in Computer Science, vol 7474. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-34091-8_3
Download citation
DOI: https://doi.org/10.1007/978-3-642-34091-8_3
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-34090-1
Online ISBN: 978-3-642-34091-8
eBook Packages: Computer ScienceComputer Science (R0)