Deep-Sarsa Based Multi-UAV Path Planning and Obstacle Avoidance in a Dynamic Environment

  • Wei Luo
  • Qirong TangEmail author
  • Changhong Fu
  • Peter Eberhard
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10942)


This study presents a Deep-Sarsa based path planning and obstacle avoidance method for unmanned aerial vehicles (UAVs). Deep-Sarsa is an on-policy reinforcement learning approach, which gains information and rewards from the environment and helps UAV to avoid moving obstacles as well as finds a path to a target based on a deep neural network. It has a significant advantage over dynamic environment compared to other algorithms. In this paper, a Deep-Sarsa model is trained in a grid environment and then deployed in an environment in ROS-Gazebo for UAVs. The experimental results show that the trained Deep-Sarsa model can guide the UAVs to the target without any collisions. This is the first time that Deep-Sarsa has been developed to achieve autonomous path planning and obstacle avoidance of UAVs in a dynamic environment.


UAV Deep-Sarsa Multi-agent Dynamic environment 



This work is supported by the project of National Natural Science Foundation of China (No. 61603277), the 13th-Five-Year-Plan on Common Technology, key project (No. 41412050101), and the Shanghai Aerospace Science and Technology Innovation Fund (SAST 2016017). Meanwhile, this work is also partially supported by the Youth 1000 program project (No. 1000231901), as well as by the Key Basic Research Project of Shanghai Science and Technology Innovation Plan (No. 15JC1403300). All these supports are highly appreciated.


  1. 1.
    Gan, S.K., Sukkarieh, S.: Multi-UAV target search using explicit decentralized gradient-based negotiation. In: IEEE International Conference on Robotics and Automation (ICRA), Shanghai, China, pp. 751–756 (2011)Google Scholar
  2. 2.
    Fu, C., Carrio, A., Campoy, P.: Efficient visual odometry and mapping for unmanned aerial vehicle using ARM-based stereo vision pre-processing system. In: International Conference on Unmanned Aircraft Systems (ICUAS), Colorado, USA, pp. 957–962 (2015)Google Scholar
  3. 3.
    Maza, I., Kondak, K., Bernard, M., Ollero, A.: Multi-UAV cooperation and control for load transportation and deployment. J. Intell. Robot. Syst. 57(1), 417–449 (2009)zbMATHGoogle Scholar
  4. 4.
    Fu, C., Carrio, A., Olivares-Mendez, M.A., Suarez-Fernandez, R., Campoy, P.: Robust real-time vision-based aircraft tracking from unmanned aerial vehicles. In: IEEE International Conference on Robotics and Automation (ICRA) (2014)Google Scholar
  5. 5.
    Hayat, S., Yanmaz, E., Brown, T.X., Bettstetter, C.: Multi-objective UAV path planning for search and rescue. In: IEEE International Conference on Robotics and Automation (ICRA), Singapore, pp. 5569–5574 (2017)Google Scholar
  6. 6.
    Sathyaraj, B.M., Jain, L.C., Finn, A., Drake, S.: Multiple UAVs path planning algorithms: a comparative study. Fuzzy Optim. Decis. Mak. 7(3), 257–267 (2008)MathSciNetCrossRefGoogle Scholar
  7. 7.
    Hrabar, S.: 3D path planning and stereo-based obstacle avoidance for rotorcraft UAVs. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, Nice, France, pp. 807–814 (2008)Google Scholar
  8. 8.
    Bounini, F., Gingras, D., Pollart, H., Gruyer, D.: Modified artificial potential field method for online path planning applications. In: IEEE Intelligent Vehicles Symposium (IV), Los Angeles, USA, pp. 180–185 (2017)Google Scholar
  9. 9.
    Galceran, E., Carreras, M.: A survey on coverage path planning for robotics. Robot. Auton. Syst. 61(12), 1258–1276 (2013)CrossRefGoogle Scholar
  10. 10.
    Zhao, Y., Zheng, Z., Zhang, X., Liu, Y.: Q learning algorithm based UAV path learning and obstacle avoidence approach. In: 36th Chinese Control Conference (CCC), Dalian, China, pp. 3397–3402 (2017)Google Scholar
  11. 11.
    Imanberdiyev, N., Fu, C., Kayacan, E., Chen, I.-M.: Autonomous navigation of UAV by using real-time model-based reinforcement learning. In: 14th International Conference on Control, Automation, Robotics and Vision, Phuket, Thailand, pp. 1–6 (2016)Google Scholar
  12. 12.
    Kubat, M.: Reinforcement learning. In: An Introduction to Machine Learning, pp. 331–339 (2017)CrossRefGoogle Scholar
  13. 13.
    Zhao, D., Wang, H., Shao, K., Zhu, Y.: Deep reinforcement learning with experience replay based on SARSA. In: IEEE Symposium Series on Computational Intelligence (SSCI) (2016)Google Scholar
  14. 14.
    Quigley, M., Conley, K., Gerkey, B., Faust, J., Foote, T., Leibs, J., Wheeler, R., Ng, A.Y.: ROS: an open-source robot operating system. In: ICRA Workshop on Open Source Software, Kobe, Japan, pp. 1–6 (2009)Google Scholar
  15. 15.
    Koenig, N., Howard, A.: Design and use paradigms for Gazebo, an open-source multi-robot simulator. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, Sendai, Japan, vol. 3, pp. 2149–2154 (2004)Google Scholar
  16. 16.
    Kober, J., Bagnell, J.A., Peters, J.: Reinforcement learning in robotics: a survey. Int. J. Robot. Res. 32(11), 1238–1274 (2013)CrossRefGoogle Scholar
  17. 17.
    Singh, S., Jaakkola, T., Littman, M.L., Szepesvári, C.: Convergence results for single-step on-policy reinforcement-learning algorithms. Mach. Learn. 38(3), 287–308 (2000). Scholar
  18. 18.
    Sutton, R.S.: Generalization in reinforcement learning: successful examples using sparse coarse coding. In: Touretzky, D.S., Mozer, M.C., Hasselmo, M.E. (eds.) Advances in Neural Information Processing Systems, pp. 1038–1044. MIT Press (1996)Google Scholar
  19. 19.
    Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A.A., Veness, J., Bellemare, M.G., Graves, A., Riedmiller, M., Fidjeland, A.K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., Hassabis, D.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015)CrossRefGoogle Scholar
  20. 20.
    Ketkar, N.: Introduction to keras. In: Deep Learning with Python, pp. 97–111 (2017)CrossRefGoogle Scholar
  21. 21.
    Huang, A.S., Olson, E., Moore, D.C.: LCM: lightweight communications and marshalling. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, Taipei, Taiwan, pp. 4057–4062 (2010)Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Wei Luo
    • 1
  • Qirong Tang
    • 2
    Email author
  • Changhong Fu
    • 2
  • Peter Eberhard
    • 1
  1. 1.Institute of Engineering and Computational MechanicsUniversity of StuttgartStuttgartGermany
  2. 2.Laboratory of Robotics and Multibody System, School of Mechanical EngineeringTongji UniversityShanghaiPeople’s Republic of China

Personalised recommendations