Signal, Image and Video Processing

, Volume 12, Issue 6, pp 1133–1140 | Cite as

Real-time road surface and semantic lane estimation using deep features

  • V. JohnEmail author
  • Z. Liu
  • S. Mita
  • C. Guo
  • K. Kidono
Original Paper


In this article, we present a robust real-time road surface and semantic lane marker estimation algorithm using the deconvolution neural network and extra trees-based decision forest. Our proposed algorithm simultaneously performs three environment perception tasks on colour and depth images, even under challenging conditions, namely road surface estimation, lane marker localization, and lane marker semantic information estimation. The lane marker semantic information implies the lane marker type such as dotted lane marker or continuous lane marker. The task of road surface estimation is performed with a trained deconvolution neural network. For the lane marker localization task, a scene-based extra trees regression framework is used to localize the lane markers in the given road. To account for the variations in the number and characteristics of the lane markers in the road scene, multiple regression models indexed with scene labels are used. The pre-defined scene labels correspond to the lane marker variations in a given scene, and an extra trees-based classification model is trained to estimate them from the road features. The road features, given as an input to the extra trees frameworks, are extracted from the road image using the trained filters of the deconvolution network. The proposed algorithm is validated using multiple acquired datasets. A comparative analysis is also conducted with baseline algorithms, and an improved accuracy is reported. Moreover, a detailed parameter evaluation is also performed. We report a computational time of 90 ms per frame.


Deep learning Intelligent vehicles Lane and road surface detection 


  1. 1.
    Adachi E, Inayoshi H, Kurita, T.: Estimation of lane state from car-mounted camera using multiple-model particle filter based on voting result for one-dimensional parameter space. In: MVA (2007)Google Scholar
  2. 2.
    Alvarez, JM., Gevers, T., Lopez, AM.: 3d scene priors for road detection. In: CVPR (2010)Google Scholar
  3. 3.
    Aly, M.: Real time detection of lane markers in urban streets. In: IVS (2008)Google Scholar
  4. 4.
    Andrew, H., Lai, S., Nelson, H., Yung, C.: Lane detection by orientation and length discrimination. SMC 30(4), 539–548 (2000)Google Scholar
  5. 5.
    Arshad, N., Moon, K., Park, S., Kim, J.: Lane detection with moving vehicle using colour information. In: World Congress on Engineering and Computer Science (2011)Google Scholar
  6. 6.
    Bertozzi, M., Broggi, A.: Gold: a parallel real-time stereo vision system for generic obstacle and lane detection. TIP 7(1), 62–81 (1998)Google Scholar
  7. 7.
    Cheng, H.Y., Jeng, B.S., Tseng, P.T., Fan, K.C.: Lane detection with moving vehicles in the traffic scenes. IEEE Trans ITS 7(4), 571–582 (2006)Google Scholar
  8. 8.
    Choi, H., Park, J., Choi, W., Oh, S.: Vision-based fusion of robust lane tracking and forward vehicle detection in a real driving environment. Int. J. Automot. Technol. 13(4), 653–669 (2012)CrossRefGoogle Scholar
  9. 9.
    Collado, JM., Hilario, C., de la Escalera, A., Armingol, JM.: Detection and classification of road lanes with a frequency analysis. In: IVS (2005)Google Scholar
  10. 10.
    El Jaafari, I., El Ansari, M., Koutti, L.: Fast edge-based stereo matching approach for road applications. Signal Image Video Process. 11(2), 267–274 (2017)CrossRefGoogle Scholar
  11. 11.
    Friedman, N., Geiger, D., Goldszmidt, M.: Bayesian network classifiers. Mach. Learn. 29(2–3), 131–163 (1997)CrossRefzbMATHGoogle Scholar
  12. 12.
    Gao, Y., Song, Y., Yang, Z.: A real-time drivable road detection algorithm in urban traffic environment. In: ICCVG (2012)Google Scholar
  13. 13.
    He, Y., Wang, H., Zhang, B.: Color-based road detection in urban traffic scenes. IEEE Trans. ITS 5(4), 309–318 (2004)Google Scholar
  14. 14.
    Huang, A.S., Teller, S.: Probabilistic lane estimation for autonomous driving using basis curves. Auton. Robot. 31(2), 269–283 (2011)CrossRefGoogle Scholar
  15. 15.
    Jia, B., Feng, W., Zhu, M.: Obstacle detection in single images with deep neural networks. Signal Image Video Process. 10(6), 1033–1040 (2016)CrossRefGoogle Scholar
  16. 16.
    Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guaddarrame, S., Darrel, T.: Caffe: Convolutional architecture for fast feature embedding. In: arXiv preprint arXiv:1408.5093 (2014)
  17. 17.
    John, V., Liu, Z., Guo, C., Mita, S., Kidono, K.: Real-time lane estimation using deep features and extra trees regression. In: PSIVT (2015)Google Scholar
  18. 18.
    John, V., Guo, C., Mita, S., Kidono, K., Guo, C., Ishimaru, K.: Fast road scene segmentation using deep learning and scene-based models. In: ICPR (2016)Google Scholar
  19. 19.
    Kim, J., Lee, M.: Robust lane detection based on convolutional neural network and random sample consensus. In: NIPS (2014)Google Scholar
  20. 20.
    Kowsari, T., Beauchemin, SS., Bauer, MA.: Map-based lane and obstacle-free area detection. In: VISAPP (2014)Google Scholar
  21. 21.
    Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: CVPR (2015)Google Scholar
  22. 22.
    Noh, H., Hong, S., Han, B.: Learning deconvolution network for semantic segmentation. CoRR abs/1505.04366 (2015)Google Scholar
  23. 23.
    Ozgunalp, U., Ai, X., Dahnoun, N.: Stereo vision-based road estimation assisted by efficient planar patch calculation. Signal Image Video Process. 10(6), 1127–1134 (2016)CrossRefGoogle Scholar
  24. 24.
    Prochazka, Z.: Road region segmentation based on sequential monte-carlo estimation. In: ICARCV (2008)Google Scholar
  25. 25.
    Protasov, S., Khan, A.M., Sozykin, K., Ahmad, M.: Using deep features for video scene detection and annotation. Signal Image Video Process. (2018).
  26. 26.
    Samadzadegan, F., Sarafraz, A., Tabibi, M.: Automatic lane detection in image sequences for vision-based navigation purpose. In: IEVM (2006)Google Scholar
  27. 27.
    Sehestedt, S., Kodagoda, S., Alempijevic, A., Dissanayake, G.: Efficient lane detection and tracking in urban environments. In: ECMR (2007)Google Scholar
  28. 28.
    Son, TT., Mita, S., Takeuchi, A.: Road detection using segmentation by weighted aggregation based on visual information and a posteriori probability of road regions. In: SMC (2008)Google Scholar
  29. 29.
    Sotelo, M.A., Rodriguez, F.J., Magdalena, L., Bergasa, L.M., Boquete, L.: A color vision-based lane tracking system for autonomous driving on unmarked roads. Auton. Robot. 16(1), 95–116 (2004)CrossRefGoogle Scholar
  30. 30.
    Southall, B., Taylor, CJ.: Stochastic road shape estimation. In: ICCV (2001)Google Scholar
  31. 31.
    Wang, Y., Shen, D., Teoh, E.K.: Lane detection using spline model. Pattern Recognit. Lett. 21(9), 677–689 (2000)CrossRefGoogle Scholar
  32. 32.
    Wu, M., Lam, S.K., Srikanthan, T.: Nonparametric technique based high-speed road surface detection. IEEE Trans. ITS 16(2), 874–884 (2015)Google Scholar
  33. 33.
    Yenikaya, S., Yenikaya, G., Düven, E.: Keeping the vehicle on the road: a survey on on-road lane detection systems. ACM Comput. Surv. 46(1), 2:1–2:43 (2013)CrossRefGoogle Scholar
  34. 34.
    Yun, S., Guo-ying, Z., Yong, Y.: A road detection algorithm by boosting using feature combination. In: IVS (2007)Google Scholar

Copyright information

© Springer-Verlag London Ltd., part of Springer Nature 2018

Authors and Affiliations

  1. 1.Toyota Technological InstituteNagoyaJapan
  2. 2.Toyota Central R & D LabsNagakuteJapan
  3. 3.University of British ColumbiaVancouverCanada

Personalised recommendations