Road Scene Risk Perception for Intelligent Vehicles Using End-to-End Affordance Learning and Visual Reasoning

  • Jayani WithanawasamEmail author
  • Ehsan Javanmardi
  • Kelvin Wong
  • Mahdi Javanmardi
  • Shunsuke Kamijo
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12046)


A key goal of intelligent vehicles is to provide a safer and more efficient method of transportation. One important aspect of intelligent vehicles is to understand the road scene using vehicle-mounted camera images. Perceiving the level of driving risk of a given road scene enables intelligent vehicles to drive more efficiently without compromising on safety. Existing road scene understanding methods, however, do not explicitly nor holistically model this notion of driving risk. This paper proposes a new perspective on scene risk perception by modeling end-to-end road scene affordance using a weakly supervised classifier. A subset of images from BDD100k dataset was relabeled to evaluate the proposed model. Experimental results show that the proposed model is able to correctly classify three different levels of risk. Further, saliency maps were used to demonstrate that the proposed model is capable of visually reasoning about the underlying causes of its decision. By understanding risk holistically, the proposed method is intended to be complementary to existing advanced driver assistance systems and autonomous vehicles.


Road scene understanding Scene understanding Road safety 


  1. 1.
    Bojarski, M., et al.: End to end learning for self-driving cars.
  2. 2.
    Chen, C., Seff, A., Kornhauser, A., Xiao, J.: DeepDriving: learning affordance for direct perception in autonomous driving. In: 2015 IEEE International Conference on Computer Vision (ICCV), pp. 2722–2730. IEEE.,
  3. 3.
    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: CVPR09 (2009)Google Scholar
  4. 4.
    Fridman, L., Brown, D.E., Kindelsberger, J., Angell, L., Mehler, B., Reimer, B.: Human side of tesla autopilot: Exploration of functional vigilance in real-world human-machine collaboration Google Scholar
  5. 5.
    Gibson, J.J.: The Ecological Approach to Visual Perception: classic edition. Psychology Press (2014)Google Scholar
  6. 6.
    Hassanin, M., Khan, S., Tahtali, M.: Visual affordance and function understanding: a survey (2018). arXiv preprint, arXiv:1807.06775
  7. 7.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015).
  8. 8.
    Jeong, S.G., Kim, J., Kim, S., Min, J.: End-to-end learning of image based lane-change decision.
  9. 9.
    Joo, J., Li, W., Steen, F.F., Zhu, S.C.: Visual persuasion: inferring communicative intents of images. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 216–223. IEEE.,
  10. 10.
    Kokubun, M., Konishi, H., Higuchi, K., Kurahashi, T., Umemura, Y., Nishi, H.: Assessment of drivers’ risk perception using a simulator. R&D Rev. Toyota CRDL 39(2), 9–15 (2004)Google Scholar
  11. 11.
    Lin, S.C., et al.: The architectural implications of autonomous driving: constraints and acceleration. In: Proceedings of the Twenty-Third International Conference on Architectural Support for Programming Languages and Operating Systems, pp. 751–766. ACM (2018)Google Scholar
  12. 12.
    Lüddecke, T., Wörgötter, F.: Scene affordance: inferring actions from household scenes. In: Workshop on Action and Anticipation for Visual Learning, European Conference on Computer Vision (2016)Google Scholar
  13. 13.
    Oliva, A.: Gist of the scene. In: Neurobiology of attention, pp. 251–256. Elsevier (2005)Google Scholar
  14. 14.
    Ramanishka, V., Chen, Y.T., Misu, T., Saenko, K.: Toward driving scene understanding: a dataset for learning driver behavior and causal reasoning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7699–7707 (2018)Google Scholar
  15. 15.
    Rasouli, A., Kotseruba, I., Tsotsos, J.K.: Are they going to cross? A benchmark dataset and baseline for pedestrian crosswalk behavior. In: 2017 IEEE International Conference on Computer Vision Workshops (ICCVW), pp. 206–213. IEEE.,
  16. 16.
    Rolison, J.J., Regev, S., Moutari, S., Feeney, A.: What are the factors that contribute to road accidents? an assessment of law enforcement views, ordinary drivers’ opinions, and road accident records. Accid. Anal. Prev. 115, 11–24 (2018)CrossRefGoogle Scholar
  17. 17.
    Sauer, A., Savinov, N., Geiger, A.: Conditional affordance learning for driving in urban environments. arXiv preprint arXiv:1806.06498 (2018)
  18. 18.
    Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-CAM: Visual explanations from deep networks via gradient-based localization.
  19. 19.
    Siam, M., Gamal, M., Abdel-Razek, M., Yogamani, S., Jagersand, M., Zhang, H.: A comparative study of real-time semantic segmentation for autonomous driving. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 587–597 (2018)Google Scholar
  20. 20.
    Sjöberg, L., Moen, B.E., Rundmo, T.: Explaining risk perception. an evaluation of the psychometric paradigm in risk perception research. Rotunde publikasjoner Rotunde 84, 55–76 (2004)Google Scholar
  21. 21.
    Wang, H.L., Cheong, L.F.: Affective understanding in film. IEEE Trans. Circuits Syst. Video Technol. 16(6), 689–704 (2006)CrossRefGoogle Scholar
  22. 22.
    Wang, S., Fidler, S., Urtasun, R.: Holistic 3d scene understanding from a single geo-tagged image. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3964–3972. IEEE.,
  23. 23.
    Xu, H., Gao, Y., Yu, F., Darrell, T.: End-to-end learning of driving models from large-scale video datasets.
  24. 24.
    Yu, F., et al.: BDD100K: A diverse driving video database with scalable annotation tooling. CoRR abs/1805.04687 (2018).
  25. 25.
    Zellers, R., Bisk, Y., Farhadi, A., Choi, Y.: From recognition to cognition: visual commonsense reasoning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6720–6731 (2019)Google Scholar
  26. 26.
    Čermák, J., Angelova, A.: Learning with proxy supervision for end-to-end visual learning. In: 2017 IEEE Intelligent Vehicles Symposium (IV), pp. 1–6 (2018)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Jayani Withanawasam
    • 1
    Email author
  • Ehsan Javanmardi
    • 2
  • Kelvin Wong
    • 2
  • Mahdi Javanmardi
    • 2
  • Shunsuke Kamijo
    • 1
    • 2
  1. 1.Department of Information and Communication EngineeringThe University of TokyoTokyoJapan
  2. 2.Institute of Industrial ScienceThe University of TokyoTokyoJapan

Personalised recommendations