Real-Time Binocular Vision Implementation on an SoC TMS320C6678 DSP

  • Rui Fan
  • Sicheng Duanmu
  • Yanan LiuEmail author
  • Yilong Zhu
  • Jianhao Jiao
  • Mohammud Junaid Bocus
  • Yang Yu
  • Lujia Wang
  • Ming Liu
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11754)


In recent years, computer binocular vision has been commonly utilized to provide depth information for autonomous vehicles. This paper presents an efficient binocular vision system implemented on an SoC TMS320C6678 DSP for real-time depth information extrapolation, where the search range propagates from the bottom of an image to its top. To further improve the stereo matching efficiency, the cost function is factorized into five independent parts. The value of each part is pre-calculated and stored in the DSP memory for direct data indexing. The experimental results illustrate that the proposed algorithm performs in real time, when processing the KITTI stereo datasets with eight cores in parallel.


Computer binocular vision Autonomous vehicles DSP 


  1. 1.
    Brink, J.A., Arenson, R.L., Grist, T.M., Lewin, J.S., Enzmann, D.: Bits and bytes: the future of radiology lies in informatics and information technology. Eur. Radiol. 27(9), 3647–3651 (2017)CrossRefGoogle Scholar
  2. 2.
    Fan, R., Ai, X., Dahnoun, N.: Road surface 3D reconstruction based on dense subpixel disparity map estimation. IEEE Trans. Image Process. 27(6), 3025–3035 (2018)MathSciNetCrossRefGoogle Scholar
  3. 3.
    Nordhoff, S.: Mobility 4.0: are consumers ready to adopt google’s self-driving car? Master’s thesis, University of Twente (2014)Google Scholar
  4. 4.
    Fan, R., Jiao, J., Ye, H., Yu, Y., Pitas, I., Liu, M.: Key ingredients of self-driving cars. arXiv preprint arXiv:1906.02939
  5. 5.
    Ozgunalp, U., Fan, R., Ai, X., Dahnoun, N.: Multiple lane detection algorithm based on novel dense vanishing point estimation. IEEE Trans. Intell. Transp. Syst. 18(3), 621–632 (2017)CrossRefGoogle Scholar
  6. 6.
    Fan, R., Dahnoun, N.: Real-time stereo vision-based lane detection system. Meas. Sci. Technol. 29(7), 074005 (2018)CrossRefGoogle Scholar
  7. 7.
    Bertozzi, M., Broggi, A.: Gold: a parallel real-time stereo vision system for generic obstacle and lane detection. IEEE Trans. Image Process. 7(1), 62–81 (1998)CrossRefGoogle Scholar
  8. 8.
    Fan, R., Wang, L., Liu, M., Pitas, I.: A robust roll angle estimation algorithm based on gradient descent. arXiv preprint arXiv:1906.01894
  9. 9.
    Fan, R., Prokhorov, V., Dahnoun, N.: Faster-than-real-time linear lane detection implementation using SoC DSP TMS320C6678. In: Proceedings IEEE International Conference Imaging Systems and Techniques (IST), pp. 306–311, October 2016Google Scholar
  10. 10.
    Boykov, Y., Veksler, O., Zabih, R.: Fast approximate energy minimization via graph cuts. IEEE Trans. Pattern Anal. Mach. Intell. 23(11), 1222–1239 (2001)CrossRefGoogle Scholar
  11. 11.
    Ihler, A.T., John III, W.F., Willsky, A.S.: Loopy belief propagation: convergence and effects of message errors. J. Mach. Learn. Res. 6, 905–936 (2005)MathSciNetzbMATHGoogle Scholar
  12. 12.
    Tappen, M.F., Freeman, W.T.: Comparison of graph cuts with belief propagation for stereo, using identical MRF parameters. In: Proceedings Ninth IEEE International Conference on Computer Vision, p. 900. IEEE (2003)Google Scholar
  13. 13.
    Mozerov, M.G., van de Weijer, J.: Accurate stereo matching by two-step energy minimization. IEEE Trans. Image Process. 24(3), 1153–1163 (2015)MathSciNetCrossRefGoogle Scholar
  14. 14.
    Sinha, S.N., Scharstein, D., Szeliski, R.: Efficient high-resolution stereo matching using local plane sweeps. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1582–1589 (2014)Google Scholar
  15. 15.
    Bleyer, M., Rhemann, C., Rother, C.: Extracting 3D scene-consistent object proposals and depth from stereo images. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7576, pp. 467–481. Springer, Heidelberg (2012). Scholar
  16. 16.
    Šára, R.: Finding the largest unambiguous component of stereo matching. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002. LNCS, vol. 2352, pp. 900–914. Springer, Heidelberg (2002). Scholar
  17. 17.
    Sara, R.: Robust correspondence recognition for computer vision. In: Rizzi, A., Vichi, M. (eds.) COMPSTAT 2006-Proceedings in Computational Statistics, pp. 119–131. Springer, Heidelberg (2006). Scholar
  18. 18.
    Cech, J., Sara, R.: Efficient sampling of disparity space for fast and accurate matching. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2007, pp. 1–8. IEEE (2007)Google Scholar
  19. 19.
    Spangenberg, R., Langner, T., Rojas, R.: Weighted semi-global matching and center-symmetric census transform for robust driver assistance. In: Wilson, R., Hancock, E., Bors, A., Smith, W. (eds.) CAIP 2013. LNCS, vol. 8048, pp. 34–41. Springer, Heidelberg (2013). Scholar
  20. 20.
    Miksik, O., Amar, Y., Vineet, V., Pérez, P., Torr, P.H.: Incremental dense multi-modal 3D scene reconstruction. In: 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 908–915. IEEE (2015)Google Scholar
  21. 21.
    Pillai, S., Ramalingam, S., Leonard, J.J.: High-performance and tunable stereo reconstruction. In: 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 3188–3195. IEEE (2016)Google Scholar
  22. 22.
    Fan, R., Liu, Y., Bocus, M.J., Wang, L., Liu, M.: Real-time subpixel fast bilateral stereo. In: 2018 IEEE International Conference on Information and Automation (ICIA), pp. 1058–1065. IEEE, August 2018Google Scholar
  23. 23.
    Fan, R., Jiao, J., Pan, J., Huang, H., Shen, S., Liu, M.: Real-time dense stereo embedded in a UAV for road inspection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (2019)Google Scholar
  24. 24.
    Zhang, Z.: Advanced stereo vision disparity calculation and obstacle analysis for intelligent vehicles. Ph.D. dissertation, University of Bristol (2013)Google Scholar
  25. 25.
    Fan, R., Liu, Y., Yang, X., Bocus, M.J., Dahnoun, N., Tancock, S.: Real-time stereo vision for road surface 3-D reconstruction. In: Proceedings of IEEE International Conference Imaging Systems and Techniques (IST), pp. 1–6, October 2018Google Scholar
  26. 26.
    Ai, X.: Active based range measurement systems and applications. Ph.D. dissertation, University of Bristol (2014)Google Scholar
  27. 27.
    Fan, R., Dahnoun, N.: Real-time implementation of stereo vision based on optimised normalised cross-correlation and propagated search range on a GPU. In: Proceedings of IEEE International Conference Imaging Systems and Techniques (IST), pp. 1–6, October 2017Google Scholar
  28. 28.
    Mano, M.M.: Computer System Architecture (2003)Google Scholar
  29. 29.
    Fan, R.: Real-time computer stereo vision for automotive applications. Ph.D. dissertation, University of Bristol, July 2018Google Scholar
  30. 30.
    Texas Instruments: Multicore fixed and floating-point digital signal processor. Literature Number SPRS691E (2014)Google Scholar
  31. 31.
    Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving? The KITTI vision benchmark suite. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 3354–3361, June 2012Google Scholar
  32. 32.
    Menze, M., Heipke, C., Geiger, A.: Joint 3D estimation of vehicles and scene flow. In: ISPRS Workshop on Image Sequence Analysis (ISA), vol. 8 (2015)CrossRefGoogle Scholar
  33. 33.
    Menze, M., Geiger, A., Heipke, C.: Object scene flow. ISPRS J. Photogram. Remote Sens. (JPRS) 140, 60–76 (2018)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Rui Fan
    • 1
  • Sicheng Duanmu
    • 2
  • Yanan Liu
    • 3
    Email author
  • Yilong Zhu
    • 4
  • Jianhao Jiao
    • 1
  • Mohammud Junaid Bocus
    • 3
  • Yang Yu
    • 1
  • Lujia Wang
    • 5
  • Ming Liu
    • 1
  1. 1.The Hong Kong University of Science and TechnologyKowloonHong Kong
  2. 2.China Unionpay Data Services Co., Ltd.ShanghaiChina
  3. 3.University of BristolBristolUK
  4. 4.Unity DriveShenzhenChina
  5. 5.Shenzhen Institutes of Advanced TechnologyChinese Academy of SciencesShenzhenChina

Personalised recommendations