Unmanned Aerial Vehicles Path Planning Based on Deep Reinforcement Learning

  • Guoqiu Wang
  • Xuanyu Zheng
  • Haitong Zhao
  • Qidong Zhao
  • Changsheng ZhangEmail author
  • Bin Zhang
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 1074)


Obstacle avoidance and path planning of unmanned aerial vehicles (UAVs) is an essential and challenging task, especially in the unknown environment with dynamic obstacles. To address this problem, a method of UAV path planning based on Deep Q-Learning is proposed. The experience replay mechanism is introduced in the deep reinforcement learning (DRL) process, and a value network is established to calculate the optimal value for the action of the UAV. The optimal flight policy of the UAV is determined through the \(\epsilon \)-greed algorithm. The experimental results show that the UAV with well-trained model can avoid the obstacles in motion perfectly, and the cruise time is reduced by half compared with the untrained UAV.


UAV path planning Obstacle avoidance Deep reinforcement learning DQN 


  1. 1.
    Dorling, K., Heinrichs, J., Messier, G.G.: Vehicle routing problems for drone delivery. IEEE Trans. Syst. Man Cybern. 47(1), 70–85 (2017). Scholar
  2. 2.
    Yin, C., Xiao, Z., Cao, X., et al.: Offline and online search: UAV multiobjective path planning under dynamic urban environment. IEEE Internet Things J. 5(2), 546–558 (2018). Scholar
  3. 3.
    Bevacqua, G., Cacace, J., Finzi, A., Lippiello, V.: Mixed-initiative planning and execution for multiple drones in search and rescue missions. In: Twenty-Fifth International Conference on Automated Planning and Scheduling. AAAI Publications (2015)Google Scholar
  4. 4.
    Chen, Y.B., Luo, G.C., Mei, Y.S., et al.: UAV path planning using artificial potential field method updated by optimal control theory. Int. J. Syst. Sci. 47(6), 1407–1420 (2016). Scholar
  5. 5.
    Bortoff, S.A.: Path Planning for UAV. In: Proceedings of the 2000 American Control Conference. ACC (IEEE Cat. No. 00CH36334) (2000)Google Scholar
  6. 6.
    Peng, X., Xu, D., Zhang, F.: UAV online path planning based on dynamic multiobjective evolutionary algorithm. In: Proceedings of the 30th Chinese Control Conference, pp. 5424–5429. IEEE (2011)Google Scholar
  7. 7.
    Ragi, S., Chong, E.K.P.: UAV path planning in a dynamic environment via partially observable Markov decision process. IEEE Trans. Aerosp. Electron. Syst. 49(4), 2397–2412 (2013). Scholar
  8. 8.
    Roberge, V., Tarbouchi, M., Labonte, G.: Comparison of parallel genetic algorithm and particle swarm optimization for real-time UAV path planning. IEEE Trans. Industr. Inf. 9(1), 132–141 (2013). Scholar
  9. 9.
    Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A.A., Veness, J., Bellemare, M.G., Graves, A., Riedmiller, M., Fidjeland, A.K., Ostrovski, G., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015). Scholar
  10. 10.
    Chen, Y.F., Liu, M., Everett, M., et al.: Decentralized non-communicating multiagent collision avoidance with deep reinforcement learning. In: 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 285–292. IEEE (2017).
  11. 11.
    Chen, C., Liu, Y., Kreiss, S., et al.: Crowd-robot interaction: crowd-aware robot navigation with attention-based deep reinforcement learning. arXiv preprint arXiv:1809.08835 (2018)
  12. 12.
    Wan, Q., Liu, W., Xu, L., et al.: Research on optimal strategy planning of uncertain environment BDI agent based on Q-learning. Comput. Eng. Sci. 41(01), 166–172 (2019)Google Scholar
  13. 13.
    Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., Wierstra, D.: Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 (2015)
  14. 14.
    Nair, A., McGrew, B., Andrychowicz, M., et al.: Overcoming exploration in reinforcement learning with demonstrations. In: 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 6292–6299. IEEE (2018)Google Scholar
  15. 15.
    Hu, Y., Da, Q., Zeng, A., et al.: Reinforcement learning to rank in e-commerce search engine: formalization, analysis, and application. In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 368–377. ACM (2018)Google Scholar
  16. 16.
    Luo, W., Tang, Q., Fu, C., Eberhard, P.: Deep-Sarsa based multi-UAV path planning and obstacle avoidance in a dynamic environment. In: International Conference on Sensing and Imaging. Advances in Swarm Intelligence, pp. 102–111 (2018).
  17. 17.
    van den Berg, J., Guy, S.J., Lin, M., Manocha, D.: Reciprocal \(n\)-body collision avoidance. In: Robotics Research. Springer Tracts in Advanced Robotics, vol. 70, pp. 3–19 (2011).

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Guoqiu Wang
    • 1
  • Xuanyu Zheng
    • 1
  • Haitong Zhao
    • 1
  • Qidong Zhao
    • 1
  • Changsheng Zhang
    • 1
    Email author
  • Bin Zhang
    • 1
  1. 1.Northeastern UniversityShenyangPeople’s Republic of China

Personalised recommendations