Cue and Sensor Fusion for Independent Moving Objects Detection and Description in Driving Scenes

  • Nikolay Chumerin
  • Marc M. Van Hulle

In this study we present an approach to detecting, describing and tracking independently moving objects (IMOs) in stereo video sequences acquired by on-board cameras on a moving vehicle. In the proposed model only three sensors are used: stereovision, speedometer and light detection and ranging (LIDAR). The IMOs detected by vision are matched with obstacles provided by LIDAR. In the case of a successful matching, the descriptions of the IMOs (distance, relative speed and acceleration) are provided by adaptive cruise control (ACC) LIDAR sensor, or otherwise these descriptions are estimated based on vision. Absolute speed of the IMO is evaluated using its relative velocity and ego-speed provided by the speedometer. Preliminary results indicate the generalization ability of the proposed system.


Ground Plane Sensor Fusion Obstacle Detection Convolutional Layer Lidar Sensor 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Beaton, A., Tukey, J.: The fitting of power series, meaning polynomials, illustrated on band-spectroscopic data. Technometrics 16(2), 147–185 (1974)MATHCrossRefGoogle Scholar
  2. 2.
    Becker, J., Simon, A.: Sensor and navigation data fusion for an autonomous vehicle. Intelligent Vehicles Symposium, 2000. IV 2000. Proceedings of the IEEE pp. 156–161 (2000)Google Scholar
  3. 3.
    Bertozzi, M., Broggi, A., Cellario, M., Fascioli, A., Lombardi, P., Porta, M.: Artificial vision in road vehicles. Proceedings of the IEEE 90(7), 1258–1271 (2002)CrossRefGoogle Scholar
  4. 4.
    Bertozzi, M., Broggi, A., Fascioli, A.: Vision-based intelligent vehicles: State of the art and perspectives. Robotics and Autonomous Systems 32(1), 1–16 (2000)CrossRefGoogle Scholar
  5. 5.
    Blanc, C., Trassoudaine, L., Le Guilloux, Y., Moreira, R.: Track to track fusion method applied to road obstacle detection. Procedings of the Seventh International Conference on Information Fusion (2004)Google Scholar
  6. 6.
    Chumerin, N., Van Hulle, M.: An approach to on-road vehicle detection, description and tracking. Proceedings of the 2007 17th IEEE Signal Processing Society Workshop on Machine Learning for Signal Processing (2007). (in press)Google Scholar
  7. 7.
    Dasarathy, B.: Sensor fusion potential exploitation-innovative architectures and illustrative applications. Proceedings of the IEEE 85(1), 24–38 (1997)CrossRefGoogle Scholar
  8. 8.
    Fang, Y., Masaki, I., Horn, B.: Depth-based target segmentation for intelligent vehicles: fusion of radar and binocular stereo. Intelligent Transportation Systems, IEEE Transactions on 3(3), 196–202 (2002)Google Scholar
  9. 9.
    Gandhi, T., Trivedi, M.: Vehicle surround capture: Survey of techniques and a novel omni video based approach for dynamic panoramic surround maps. Intelligent Transportation Systems, IEEE Transactions on 7(3), 293–308 (2006)Google Scholar
  10. 10.
    Hall, D., Llinas, J.: An introduction to multisensor data fusion. Proceedings of the IEEE 85(1), 6–23 (1997)CrossRefGoogle Scholar
  11. 11.
    Handmann, U., Lorenz, G., Schnitger, T., Seelen, W.: Fusion of different sensors and algorithms for segmentation. IV’98, IEEE International Conference on Intelligent Vehicles 1998 pp. 499–504 (1998)Google Scholar
  12. 12.
    Hofmann, U., Rieder, A., Dickmanns, E.: Radar and vision data fusion for hybrid adaptive cruise control on highways. Machine Vision and Applications 14(1), 42–49 (2003)CrossRefGoogle Scholar
  13. 13.
    Julier, S., Uhlmann, J.: A new extension of the kalman filter to nonlinear systems. International Symposium Aerospace/Defense Sensing, Simulation and Controls 3 (1997)Google Scholar
  14. 14.
    Kalman, R.: A new approach to linear filtering and prediction problems. Journal of Basic Engineering 82(1), 35–45 (1960)Google Scholar
  15. 15.
    Kastrinaki, V., Zervakis, M., Kalaitzakis, K.: A survey of video processing techniques for traffic applications. Image and Vision Computing 21(4), 359–381 (2003)CrossRefGoogle Scholar
  16. 16.
    Kato, T., Ninomiya, Y., Masaki, I.: An obstacle detection method by fusion of radar and motion stereo. Intelligent Transportation Systems, IEEE Transactions on 3(3), 182–188 (2002)Google Scholar
  17. 17.
    Labayrade, R., Royere, C., Gruyer, D., Aubert, D.: Cooperative fusion for multi-obstacles detection with use of stereovision and laser scanner. Autonomous Robots 19(2), 117–140 (2005)CrossRefGoogle Scholar
  18. 18.
    Laneurit, J., Blanc, C., Chapuis, R., Trassoudaine, L.: Multisensorial data fusion for global vehicle and obstacles absolute positioning. Intelligent Vehicles Symposium, 2003. Proceedings. IEEE pp. 138–143 (2003)Google Scholar
  19. 19.
    LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. pp. 2278–2324 (1998)Google Scholar
  20. 20.
    Leibe, B., Schiele, B.: Scale invariant object categorization using a scale-adaptive mean-shift search. DAGM’04 pp. 145–153 (2004)Google Scholar
  21. 21.
    van der Merwe, R.: Sigma-point kalman filters for probabilistic inference in dynamic state-space models. Ph.D. thesis, University of Stellenbosch (2004)Google Scholar
  22. 22.
    Pauwels, K., Van Hulle, M.: Segmenting independently moving objects from egomotion flow fields. Isle of Skye, Scotland (2004)Google Scholar
  23. 23.
    Pauwels, K., Van Hulle, M.: Optic flow from unstable sequences containing unconstrained scenes through local velocity constancy maximization. pp. 397–406. Edinburgh (2006)Google Scholar
  24. 24.
    Pohl, C.: Review article multisensor image fusion in remote sensing: concepts, methods and applications. International Journal of Remote Sensing 19(5), 823–854 (1998)CrossRefMathSciNetGoogle Scholar
  25. 25.
    Sabatini, S., Gastaldi, G., Solari, F., Diaz, J., Ros, E., Pauwels, K., Van Hulle, M., Pugeault, N., Krueger, N.: Compact and accurate early vision processing in the harmonic space. Barcelona (2007)Google Scholar
  26. 26.
    Sergi, M.: Bus rapid transit technologies: A virtual mirror for eliminating vehicle blind zones. University of Minnesota ITS Institute Final Report (2003)Google Scholar
  27. 27.
    Sole, A., Mano, O., Stein, G., Kumon, H., Tamatsu, Y., Shashua, A.: Solid or not solid: vision for radar target validation. Intelligent Vehicles Symposium, 2004 IEEE pp. 819–824 (2004)Google Scholar
  28. 28.
    Steux, B., Laurgeau, C., Salesse, L., Wautier, D.: Fade: a vehicle detection and tracking system featuring monocular color vision and radar data fusion. Intelligent Vehicle Symposium, 2002. IEEE 2 (2002)Google Scholar
  29. 29.
    Stiller, C., Hipp, J., Rossig, C., Ewald, A.: Multisensor obstacle detection and tracking. Image and Vision Computing 18(5), 389–396 (2000)CrossRefGoogle Scholar
  30. 30.
    Sun, Z., Bebis, G., Miller, R.: On-road vehicle detection: a review. Pattern Analysis and Machine Intelligence, IEEE Transactions on 28(5), 694–711 (2006)Google Scholar
  31. 31.
    Thrun, S., Montemerlo, M., Dahlkamp, H., Stavens, D., Aron, A., Diebel, J., Fong, P., Gale, J., Halpenny, M., Hoffmann, G., et al.: Stanley: The robot that won the darpa grand challenge. Journal of Field Robotics 23(9), 661–692 (2006)CrossRefGoogle Scholar
  32. 32.
    Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. Proceedings of CVPR 1, 511–518 (2001)Google Scholar
  33. 33.
    Wald, L.: Some terms of reference in data fusion. IEEE Transactions on Geoscience and Remote Sensing 37(3) (1999)Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2008

Authors and Affiliations

  • Nikolay Chumerin
    • 1
  • Marc M. Van Hulle
    • 1
  1. 1.Katholieke Universiteit LeuvenBelgium

Personalised recommendations