Safe and Goal-Based Highway Maneuver Planning with Reinforcement Learning

Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 1196)


As autonomous driving moves closer to a real-world application, more and more attention is being paid to the motion planning part of the system. To handle vastness of possible road scenarios, negotiate with other road users and generate an intelligent control strategy in a constantly changing environment, data-driven techniques and artificial intelligence methods seem to be the approach of choice. In this paper, we present reinforcement learning (RL) agent which is embedded in a deterministic, safety envelope. The agent is responsible for generating high-level maneuvers, such as a lane following or a lane change. The primary goal of the agent is to reach a given lane in a given distance, while traveling on a highway. The selected maneuver is then executed with use of deterministic methods utilizing concept of Responsible-Sensitive Safety (RSS) framework, which formalizes safety constrains in a form of mathematical model. The proposed solution has been evaluated in two environments: one in which the agent receives a predefined reward for getting to a correct lane and second, in which it is rewarded for doing this in a time-optimal manner. We have evaluated the proposed solution against an another RL-based agent, which is steering vehicle by low-level control signals, such as acceleration and steering angle.


Autonomous driving Behavior planning Maneuver planning Deep reinforcement learning 


  1. 1.
    Althoff, M., Stursberg, O., Buss, M.: Model-based probabilistic collision detection in autonomous driving. IEEE Trans. Intell. Transp. Syst. 10(2), 299–310 (2009).
  2. 2.
    Bojarski, M., Del Testa, D., Dworakowski, D., Firner, B., Flepp, B., Goyal, P., Jackel, L.D., Monfort, M., Muller, U., Zhang, J., Zhang, X., Zhao, J., Zieba, K., et al.: End to End Learning for Self-Driving Cars (2016).
  3. 3.
    Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., Zaremba, W.: OpenAI Gym. CoRR abs/1606.01540 (2016).
  4. 4.
    Censi, A., Slutsky, K., Wongpiromsarn, T., Yershov, D., Pendleton, S., Fu, J., Frazzoli, E.: Liability, Ethics, and Culture-Aware Behavior Specification using Rulebooks, pp. 8536–8542. Institute of Electrical and Electronics Engineers (IEEE) (2019).
  5. 5.
    Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., Koltun, V.: CARLA: An Open Urban Driving Simulator (2017). URL
  6. 6.
    Haarnoja, T., Zhou, A., Abbeel, P., Levine, S.: Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor (2018).
  7. 7.
    Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., Levine, S.: Soft Actor-Critic Algorithms and Applications (2018).
  8. 8.
    Hausknecht, M., Stone, P.: Deep recurrent q-learning for partially observable MDPs. In: AAAI Fall Symposium - Technical Report, vol. FS-15-06, pp. 29–37. AI Access Foundation (2015)Google Scholar
  9. 9.
    Horgan, D., Quan, J., Budden, D., Barth-Maron, G., Hessel, M., van Hasselt, H., Silver, D.: Distributed prioritized experience replay. In: International Conference on Learning Representations (2018).
  10. 10.
    Kapturowski, S., Ostrovski, G., Dabney, W., Quan, J., Munos, R.: Recurrent experience replay in distributed reinforcement learning. In: International Conference on Learning Representations (2019).
  11. 11.
    Lopez, P.A., Behrisch, M., Bieker-Walz, L., Erdmann, J., Flotterod, Y.P., Hilbrich, R., Lucken, L., Rummel, J., Wagner, P., Wiebner, E.: Microscopic Traffic Simulation using SUMO. In: IEEE Conference on Intelligent Transportation Systems, Proceedings, ITSC, vol. 2018-November, pp. 2575–2582. Institute of Electrical and Electronics Engineers Inc. (2018).
  12. 12.
    Magdici, S., Althoff, M.: Fail-safe motion planning of autonomous vehicles. In: 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC), pp. 452–458. IEEE, Rio de Janeiro, Brazil (2016).
  13. 13.
    Makantasis, K., Kontorinaki, M., Nikolos, I.: A Deep Reinforcement-Learning-based Driving Policy for Autonomous Road Vehicles (2019).
  14. 14.
    Mirchevska, B., Pek, C., Werling, M., Althoff, M., Boedecker, J.: High-level decision making for safe and reasonable autonomous lane changing using reinforcement learning. In: IEEE Conference on Intelligent Transportation Systems, Proceedings, ITSC, vol. 2018-November, pp. 2156–2162. Institute of Electrical and Electronics Engineers Inc. (2018).
  15. 15.
    Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., Riedmiller, M.: Playing Atari with Deep Reinforcement Learning (2013).
  16. 16.
    Nageshrao, S., Tseng, E., Filev, D.: Autonomous Highway Driving using Deep Reinforcement Learning (2019).
  17. 17.
    Ngai, D.C.K., Yung, N.H.C.: A multiple-goal reinforcement learning method for complex vehicle overtaking maneuvers. IEEE Trans. Intell. Transp. Syst. 12, 509–522 (2011). Scholar
  18. 18.
    Nistér, D., Lee, H.L., Ng, J., Wang, Y.: The Safety Force Field. Technical reportGoogle Scholar
  19. 19.
    Paden, B., Cap, M., Yong, S.Z., Yershov, D., Frazzoli, E.: A survey of motion planning and control techniques for self-driving urban vehicles. IEEE Trans. Intell. Veh. 1(1), 33–55 (2016). Scholar
  20. 20.
    Qian, X., de La Fortelle, A., Moutarde, F.: A hierarchical Model Predictive Control framework for on-road formation control of autonomous vehicles. In: 2016 IEEE Intelligent Vehicles Symposium (IV), pp. 376–381. IEEE, Gotenburg, Sweden (2016).
  21. 21.
    Rasekhipour, Y., Khajepour, A., Chen, S.K., Litkouhi, B.: A potential field-based model predictive path-planning controller for autonomous road vehicles. IEEE Trans. Intell. Transp. Syst. 18(5), 1255–1267 (2017).
  22. 22.
    Schaul, T., Quan, J., Antonoglou, I., Silver, D.: Prioritized experience replay. arXiv preprint arXiv:1511.05952 (2015)
  23. 23.
    Shah, S., Dey, D., Lovett, C., Kapoor, A.: AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles (2017).
  24. 24.
    Shalev-Shwartz, S., Shammah, S., Shashua, A.: On a Formal Model of Safe and Scalable Self-driving Cars (2017).
  25. 25.
    Sutton, R.S., Barto, A.G.: Introduction to Reinforcement Learning, 1st edn. MIT Press, Cambridge (1998)zbMATHGoogle Scholar
  26. 26.
    Talpaert, V., Sobh, I., Ravi Kiran, B., Mannion, P., Yogamani, S., El-Sallab, A., Perez, P.: Exploring applications of deep reinforcement learning for real-world autonomous driving systems. In: VISIGRAPP 2019 - Proceedings of the 14th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, vol. 5, pp. 564–572. SciTePress (2019).
  27. 27.
    Wang, P., Chan, C.Y.: Formulation of deep reinforcement learning architecture toward autonomous driving for on-ramp merge. In: IEEE Conference on Intelligent Transportation Systems, Proceedings, ITSC, vol. 2018-March, pp. 1–6. Institute of Electrical and Electronics Engineers Inc. (2018).
  28. 28.
    Wei, J., Snider, J.M., Gu, T., Dolan, J.M., Litkouhi, B.: A behavioral planning framework for autonomous driving. In: IEEE Intelligent Vehicles Symposium, Proceedings, pp. 458–464. Institute of Electrical and Electronics Engineers Inc. (2014).
  29. 29.
    Xu, W., Wei, J., Dolan, J.M., Zhao, H., Zha, H.: A real-time motion planner with trajectory optimization for autonomous vehicles. In: 2012 IEEE International Conference on Robotics and Automation, pp. 2061–2067. IEEE (2012)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.AGH University of Science and TechnologyKrakowPoland
  2. 2.AptivKrakowPoland

Personalised recommendations