An Evaluation Framework for Stereo-Based Driver Assistance

  • Nicolai Schneider
  • Stefan Gehrig
  • David Pfeiffer
  • Konstantinos Banitsas
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7474)


The accuracy of stereo algorithms or optical flow methods is commonly assessed by comparing the results against the Middlebury database. However, equivalent data for automotive or robotics applications rarely exist as they are difficult to obtain. As our main contribution, we introduce an evaluation framework tailored for stereo-based driver assistance able to deliver excellent performance measures while circumventing manual label effort. Within this framework one can combine several ways of ground-truthing, different comparison metrics, and use large image databases.

Using our framework we show examples on several types of ground-truthing techniques: implicit ground truthing (e.g. sequence recorded without a crash occurred), robotic vehicles with high precision sensors, and to a small extent, manual labeling. To show the effectiveness of our evaluation framework we compare three different stereo algorithms on pixel and object level. In more detail we evaluate an intermediate representation called the Stixel World. Besides evaluating the accuracy of the Stixels, we investigate the completeness (equivalent to the detection rate) of the Stixel World vs. the number of phantom Stixels. Among many findings, using this framework enables us to reduce the number of phantom Stixels by a factor of three compared to the base parametrization. This base parametrization has already been optimized by test driving vehicles for distances exceeding 10000 km.


Ground Truth Evaluation Framework Velocity Error Ground Truth Data Stereo Match 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Badino, H.: A Robust Approach for Ego-Motion Estimation Using a Mobile Stereo Platform. In: Jähne, B., Mester, R., Barth, E., Scharr, H. (eds.) IWCM 2004. LNCS, vol. 3417, pp. 198–208. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  2. 2.
    Badino, H., Franke, U., Mester, R.: Free space computation using stochastic occupancy grids and dynamic programming. In: Workshop on Dynamical Vision, ICCV, Rio de Janeiro, Brazil (October 2007)Google Scholar
  3. 3.
    Badino, H., Franke, U., Pfeiffer, D.: The Stixel World - A Compact Medium Level Representation of the 3D-World. In: Denzler, J., Notni, G., Süße, H. (eds.) DAGM 2009. LNCS, vol. 5748, pp. 51–60. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  4. 4.
    Barth, A.: Vehicle Tracking and Motion Estimation Based on Stereo Vision Sequences. PhD thesis, Friedrich-Wilhelms-Universitaet zu Bonn (September 2010)Google Scholar
  5. 5.
    Barth, A., Franke, U.: Where will the oncoming vehicle be the next second? In: IEEE Intelligent Vehicles Symposium (IV), Eindhoven, Netherlands, pp. 1068–1073 (April 2008)Google Scholar
  6. 6.
    Barth, A., Siegemund, J., Franke, U., Förstner, W.: Simultaneous Estimation of Pose and Motion at Highly Dynamic Turn Maneuvers. In: Denzler, J., Notni, G., Süße, H. (eds.) DAGM 2009. LNCS, vol. 5748, pp. 262–271. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  7. 7.
    Brox, T., Weickert, J.: Nonlinear Matrix Diffusion for Optic Flow Estimation. In: Van Gool, L. (ed.) DAGM 2002. LNCS, vol. 2449, pp. 446–453. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  8. 8.
    Collins, R., Tsin, Y., Miller, J.R., Lipton, A.: Using a dem to determine geospatial object trajectories. In: Proceedings of the 1998 DARPA Image Understanding Workshop, pp. 115–122 (1998)Google Scholar
  9. 9.
    Courtney, P., Thacker, N., Clark, A.: Algorithmic modeling for performance evaluation. In: IAPR Conference on Machine Vision Applications (MVA), pp. 219–228 (1997)Google Scholar
  10. 10.
    Dreuw, P., Steingrube, P., Deselaers, T., Ney, H.: Smoothed Disparity Maps for Continuous American Sign Language Recognition. In: Araujo, H., Mendonça, A.M., Pinho, A.J., Torres, M.I. (eds.) IbPRIA 2009. LNCS, vol. 5524, pp. 24–31. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  11. 11.
    Duffy, B.R., Garcia, C., Rooney, C.F.B., O’Hare, G.M.P.: Sensor fusion for social robotics. In: 31st International Symposium on Robotics, pp. 155–170 (2000)Google Scholar
  12. 12.
    Everingham, M., Zisserman, A., Williams, C.K.I., Van Gool, L.: The 2005 pascal visual object classes challenge. Selected Proceedings of the 1st PASCAL Challenges Workshop, Springer (2006)Google Scholar
  13. 13.
    Franke, U.: Real-time stereo vision for urban traffic scene understanding. In: IEEE Intelligent Vehicles Symposium, IV (2000)Google Scholar
  14. 14.
    Franke, U., Rabe, C., Badino, H., Gehrig, S.: 6D-Vision: Fusion of Stereo and Motion for Robust Environment Perception. In: Kropatsch, W.G., Sablatnig, R., Hanbury, A. (eds.) DAGM 2005. LNCS, vol. 3663, pp. 216–223. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  15. 15.
    Fraundorfer, F., Scaramuzza, D., Pollefeys, M.: A constricted bundle adjustment parameterization for relative scale estimation in visual odometry. In: IEEE International Conference on Robotics and Automation (ICRA), Anchorage, Alaska, USA, pp. 1899–1904 (May 2010)Google Scholar
  16. 16.
    Gehrig, S.K., Eberli, F., Meyer, T.: A Real-Time Low-Power Stereo Vision Engine Using Semi-Global Matching. In: Fritz, M., Schiele, B., Piater, J.H. (eds.) ICVS 2009. LNCS, vol. 5815, pp. 134–143. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  17. 17.
    Hohm, A., Wojek, C., Bernt, S., Winner, H.: Multi level sensorfusion and computer-vision algorithms within a driver assistance system for avoiding overtaking accidents. In: FISITA World Automotive Congress, pp. 1–14 (2008)Google Scholar
  18. 18.
    Hong, T., Chang, T., Takeuchi, A., Cheok, G., Scott, H., Shneier, M.: Performance evaluation of sensors on mobile vehicles using a large data repository and ground truth. In: Proceedings PerMIS (2003)Google Scholar
  19. 19.
    Huang, W., Tan, C.-L., Zhao, J.: Generating Ground Truthed Dataset of Chart Images: Automatic or Semi-automatic? In: Liu, W., Lladós, J., Ogier, J.-M. (eds.) GREC 2007. LNCS, vol. 5046, pp. 266–277. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  20. 20.
    iMAR Navigation. iTraceRT-F200 (August 2011),
  21. 21.
    Kitt, B., Geiger, A., Lategahn, H.: Visual odometry based on stereo image sequences with ransac-based outlier rejection scheme. In: IEEE Intelligent Vehicles Symposium (IV), San Diego, CA, USA, pp. 486–492 (June 2010)Google Scholar
  22. 22.
    Lemaire, T., Berger, C., Jung, I.-K., Lacroix, S.: Vision-based slam: Stereo and monocular approaches. International Journal of Computer Vision (IJCV) 74(3), 343–364 (2007)CrossRefGoogle Scholar
  23. 23.
    Levin, A., Szeliski, R.: Visual odometry and map correlation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Washington, DC, USA, pp. 611–618 (June 2004)Google Scholar
  24. 24.
    Liu, Z., Klette, R.: Approximated Ground Truth for Stereo and Motion Analysis on Real-World Sequences. In: Wada, T., Huang, F., Lin, S. (eds.) PSIVT 2009. LNCS, vol. 5414, pp. 874–885. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  25. 25.
    Lucas, B.D., Kanade, T.: An iterative image registration technique with an application to stereo vision. In: Proceedings of the Seventh International Joint Conference on Artifical Intelligence, IJCAI 1981, Vancouver, Canada, pp. 674–679 (1981)Google Scholar
  26. 26.
    Manohar, V., Soundararajan, P., Raju, H., Goldgof, D., Kasturi, R., Garofolo, J.S.: Performance Evaluation of Object Detection and Tracking in Video. In: Narayanan, P.J., Nayar, S.K., Shum, H.-Y. (eds.) ACCV 2006. LNCS, vol. 3852, pp. 151–161. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  27. 27.
    Mariano, V.Y., Min, J., Park, J.H., Kasturi, R., Mihalcik, D., Li, H., Doermann, D.S., Drayer, T.: Performance evaluation of object detection algorithms. In: International Conference on Pattern Recognition, ICPR (2002)Google Scholar
  28. 28.
    Morales, S., Vaudrey, T., Klette, R.: Robustness evaluation of stereo algorithms on long stereo sequences. In: IEEE Intelligent Vehicles Symposium (IV), pp. 347–352 (2009)Google Scholar
  29. 29.
    Müller, T., Rannacher, J., Rabe, C., Franke, U.: Feature and depth-supported modified total variation optical flow for 3d motion field estimation in real scenes. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Colorado Springs, CO, USA, pp. 1193–1200 (June 2011)Google Scholar
  30. 30.
    Pfeiffer, D., Franke, U.: Efficient representation of traffic scenes by means of dynamic Stixels. In: IEEE Intelligent Vehicles Symposium (IV), San Diego, CA, USA, pp. 217–224 (June 2010)Google Scholar
  31. 31.
    Pfeiffer, D., Morales, S., Barth, A., Franke, U.: Ground truth evaluation of the Stixel representation using laser scanners. In: IEEE Conference on Intelligent Transportation Systems (ITSC), Maideira Island, Portugal (September 2010)Google Scholar
  32. 32.
    Scharstein, D., Szeliski, R.: Middlebury online stereo evaluation (2002),
  33. 33.
    Schneider, N.: Evaluation of stereo-based scene analysis under real-world conditions. Master’s thesis, Brunel University (July 2011)Google Scholar
  34. 34.
    Continental Automotive Industrial Sensors. ARS 300 Long Range Radar Sensor 77 GHz (July 2011),
  35. 35.
    Stein, F.: Efficient Computation of Optical Flow Using the Census Transform. In: Rasmussen, C.E., Bülthoff, H.H., Schölkopf, B., Giese, M.A. (eds.) DAGM 2004. LNCS, vol. 3175, pp. 79–86. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  36. 36.
    Steingrube, P., Gehrig, S., Franke, U.: Performance Evaluation of Stereo Algorithms for Automotive Applications. In: Fritz, M., Schiele, B., Piater, J.H. (eds.) ICVS 2009. LNCS, vol. 5815, pp. 285–294. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  37. 37.
    Tech-News. Toyota’ lexus ls 460 employs stereo camera, (viewed April 15, 2009)
  38. 38.
    Tistarelli, M.: Multiple Constraints for Optical Flow. In: Eklundh, J.-O. (ed.) ECCV 1994. LNCS, vol. 800, pp. 61–70. Springer, Heidelberg (1994)CrossRefGoogle Scholar
  39. 39.
    Tomasi, C., Shi, J.: Good features to track. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Los Alamitos, CA, USA, pp. 593–600 (June 1994)Google Scholar
  40. 40.
    Vaudrey, T., Rabe, C., Klette, R., Milburn, J.: Differences between stereo and motion behaviour on synthetic and real-world stereo sequences. In: 23rd International Conference on Image and Vision Computing New Zealand, IVCNZ 2008, November 26-28, pp. 1–6 (2008)Google Scholar
  41. 41.
    Yamada, K., Mochizuki, K., Aizawa, K., Saito, T.: Motion Segmentation with Census Transform. In: Shum, H.-Y., Liao, M., Chang, S.-F. (eds.) PCM 2001. LNCS, vol. 2195, pp. 903–908. Springer, Heidelberg (2001)CrossRefGoogle Scholar
  42. 42.
    Yilmaz, A.: Sensor fusion in computer vision. In: EEE GRSS/ISPRS Joint Workshop on Remote Sensing and Data Fusion over Urban Areas (2007)Google Scholar
  43. 43.
    Zach, C., Pock, T., Bischof, H.: A Duality Based Approach for Realtime TV–L1 Optical Flow. In: Hamprecht, F.A., Schnörr, C., Jähne, B. (eds.) DAGM 2007. LNCS, vol. 4713, pp. 214–223. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  44. 44.
    Zanibbi, R., Blostein, D., Cordy, J.R.: White-box evaluation of computer vision algorithms through explicit decision-making (2009)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Nicolai Schneider
    • 1
  • Stefan Gehrig
    • 2
  • David Pfeiffer
    • 2
  • Konstantinos Banitsas
    • 3
  1. 1.IT-Designers GmbHEsslingenGermany
  2. 2.Team Image UnderstandingDaimler AGSindelfingenGermany
  3. 3.Brunel UniversityLondonUK

Personalised recommendations