Advertisement

Safe and Goal-Based Highway Maneuver Planning with Reinforcement Learning

Conference paper
  • 410 Downloads
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 1196)

Abstract

As autonomous driving moves closer to a real-world application, more and more attention is being paid to the motion planning part of the system. To handle vastness of possible road scenarios, negotiate with other road users and generate an intelligent control strategy in a constantly changing environment, data-driven techniques and artificial intelligence methods seem to be the approach of choice. In this paper, we present reinforcement learning (RL) agent which is embedded in a deterministic, safety envelope. The agent is responsible for generating high-level maneuvers, such as a lane following or a lane change. The primary goal of the agent is to reach a given lane in a given distance, while traveling on a highway. The selected maneuver is then executed with use of deterministic methods utilizing concept of Responsible-Sensitive Safety (RSS) framework, which formalizes safety constrains in a form of mathematical model. The proposed solution has been evaluated in two environments: one in which the agent receives a predefined reward for getting to a correct lane and second, in which it is rewarded for doing this in a time-optimal manner. We have evaluated the proposed solution against an another RL-based agent, which is steering vehicle by low-level control signals, such as acceleration and steering angle.

Keywords

Autonomous driving Behavior planning Maneuver planning Deep reinforcement learning 

References

  1. 1.
    Althoff, M., Stursberg, O., Buss, M.: Model-based probabilistic collision detection in autonomous driving. IEEE Trans. Intell. Transp. Syst. 10(2), 299–310 (2009).  https://doi.org/10.1109/TITS.2009.2018966. http://ieeexplore.ieee.org/document/4895669/
  2. 2.
    Bojarski, M., Del Testa, D., Dworakowski, D., Firner, B., Flepp, B., Goyal, P., Jackel, L.D., Monfort, M., Muller, U., Zhang, J., Zhang, X., Zhao, J., Zieba, K., et al.: End to End Learning for Self-Driving Cars (2016). http://arxiv.org/abs/1604.07316
  3. 3.
    Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., Zaremba, W.: OpenAI Gym. CoRR abs/1606.01540 (2016). http://arxiv.org/abs/1606.01540
  4. 4.
    Censi, A., Slutsky, K., Wongpiromsarn, T., Yershov, D., Pendleton, S., Fu, J., Frazzoli, E.: Liability, Ethics, and Culture-Aware Behavior Specification using Rulebooks, pp. 8536–8542. Institute of Electrical and Electronics Engineers (IEEE) (2019).  https://doi.org/10.1109/icra.2019.8794364
  5. 5.
    Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., Koltun, V.: CARLA: An Open Urban Driving Simulator (2017). URL http://arxiv.org/abs/1711.03938
  6. 6.
    Haarnoja, T., Zhou, A., Abbeel, P., Levine, S.: Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor (2018). http://arxiv.org/abs/1801.01290
  7. 7.
    Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., Levine, S.: Soft Actor-Critic Algorithms and Applications (2018). http://arxiv.org/abs/1812.05905
  8. 8.
    Hausknecht, M., Stone, P.: Deep recurrent q-learning for partially observable MDPs. In: AAAI Fall Symposium - Technical Report, vol. FS-15-06, pp. 29–37. AI Access Foundation (2015)Google Scholar
  9. 9.
    Horgan, D., Quan, J., Budden, D., Barth-Maron, G., Hessel, M., van Hasselt, H., Silver, D.: Distributed prioritized experience replay. In: International Conference on Learning Representations (2018). https://openreview.net/forum?id=H1Dy--0Z
  10. 10.
    Kapturowski, S., Ostrovski, G., Dabney, W., Quan, J., Munos, R.: Recurrent experience replay in distributed reinforcement learning. In: International Conference on Learning Representations (2019). https://openreview.net/forum?id=r1lyTjAqYX
  11. 11.
    Lopez, P.A., Behrisch, M., Bieker-Walz, L., Erdmann, J., Flotterod, Y.P., Hilbrich, R., Lucken, L., Rummel, J., Wagner, P., Wiebner, E.: Microscopic Traffic Simulation using SUMO. In: IEEE Conference on Intelligent Transportation Systems, Proceedings, ITSC, vol. 2018-November, pp. 2575–2582. Institute of Electrical and Electronics Engineers Inc. (2018).  https://doi.org/10.1109/ITSC.2018.8569938
  12. 12.
    Magdici, S., Althoff, M.: Fail-safe motion planning of autonomous vehicles. In: 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC), pp. 452–458. IEEE, Rio de Janeiro, Brazil (2016).  https://doi.org/10.1109/ITSC.2016.7795594. http://ieeexplore.ieee.org/document/7795594/
  13. 13.
    Makantasis, K., Kontorinaki, M., Nikolos, I.: A Deep Reinforcement-Learning-based Driving Policy for Autonomous Road Vehicles (2019). http://arxiv.org/abs/1907.05246
  14. 14.
    Mirchevska, B., Pek, C., Werling, M., Althoff, M., Boedecker, J.: High-level decision making for safe and reasonable autonomous lane changing using reinforcement learning. In: IEEE Conference on Intelligent Transportation Systems, Proceedings, ITSC, vol. 2018-November, pp. 2156–2162. Institute of Electrical and Electronics Engineers Inc. (2018).  https://doi.org/10.1109/ITSC.2018.8569448
  15. 15.
    Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., Riedmiller, M.: Playing Atari with Deep Reinforcement Learning (2013). http://arxiv.org/abs/1312.5602
  16. 16.
    Nageshrao, S., Tseng, E., Filev, D.: Autonomous Highway Driving using Deep Reinforcement Learning (2019). http://arxiv.org/abs/1904.00035
  17. 17.
    Ngai, D.C.K., Yung, N.H.C.: A multiple-goal reinforcement learning method for complex vehicle overtaking maneuvers. IEEE Trans. Intell. Transp. Syst. 12, 509–522 (2011).  https://doi.org/10.1109/TITS.2011.2106158CrossRefGoogle Scholar
  18. 18.
    Nistér, D., Lee, H.L., Ng, J., Wang, Y.: The Safety Force Field. Technical reportGoogle Scholar
  19. 19.
    Paden, B., Cap, M., Yong, S.Z., Yershov, D., Frazzoli, E.: A survey of motion planning and control techniques for self-driving urban vehicles. IEEE Trans. Intell. Veh. 1(1), 33–55 (2016).  https://doi.org/10.1109/tiv.2016.2578706CrossRefGoogle Scholar
  20. 20.
    Qian, X., de La Fortelle, A., Moutarde, F.: A hierarchical Model Predictive Control framework for on-road formation control of autonomous vehicles. In: 2016 IEEE Intelligent Vehicles Symposium (IV), pp. 376–381. IEEE, Gotenburg, Sweden (2016).  https://doi.org/10.1109/IVS.2016.7535413. http://ieeexplore.ieee.org/document/7535413/
  21. 21.
    Rasekhipour, Y., Khajepour, A., Chen, S.K., Litkouhi, B.: A potential field-based model predictive path-planning controller for autonomous road vehicles. IEEE Trans. Intell. Transp. Syst. 18(5), 1255–1267 (2017).  https://doi.org/10.1109/TITS.2016.2604240. http://ieeexplore.ieee.org/document/7576661/
  22. 22.
    Schaul, T., Quan, J., Antonoglou, I., Silver, D.: Prioritized experience replay. arXiv preprint arXiv:1511.05952 (2015)
  23. 23.
    Shah, S., Dey, D., Lovett, C., Kapoor, A.: AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles (2017). http://arxiv.org/abs/1705.05065
  24. 24.
    Shalev-Shwartz, S., Shammah, S., Shashua, A.: On a Formal Model of Safe and Scalable Self-driving Cars (2017). http://arxiv.org/abs/1708.06374
  25. 25.
    Sutton, R.S., Barto, A.G.: Introduction to Reinforcement Learning, 1st edn. MIT Press, Cambridge (1998)zbMATHGoogle Scholar
  26. 26.
    Talpaert, V., Sobh, I., Ravi Kiran, B., Mannion, P., Yogamani, S., El-Sallab, A., Perez, P.: Exploring applications of deep reinforcement learning for real-world autonomous driving systems. In: VISIGRAPP 2019 - Proceedings of the 14th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, vol. 5, pp. 564–572. SciTePress (2019).  https://doi.org/10.5220/0007520305640572
  27. 27.
    Wang, P., Chan, C.Y.: Formulation of deep reinforcement learning architecture toward autonomous driving for on-ramp merge. In: IEEE Conference on Intelligent Transportation Systems, Proceedings, ITSC, vol. 2018-March, pp. 1–6. Institute of Electrical and Electronics Engineers Inc. (2018).  https://doi.org/10.1109/ITSC.2017.8317735
  28. 28.
    Wei, J., Snider, J.M., Gu, T., Dolan, J.M., Litkouhi, B.: A behavioral planning framework for autonomous driving. In: IEEE Intelligent Vehicles Symposium, Proceedings, pp. 458–464. Institute of Electrical and Electronics Engineers Inc. (2014).  https://doi.org/10.1109/IVS.2014.6856582
  29. 29.
    Xu, W., Wei, J., Dolan, J.M., Zhao, H., Zha, H.: A real-time motion planner with trajectory optimization for autonomous vehicles. In: 2012 IEEE International Conference on Robotics and Automation, pp. 2061–2067. IEEE (2012)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.AGH University of Science and TechnologyKrakowPoland
  2. 2.AptivKrakowPoland

Personalised recommendations