Scalable Markov chain approximation for a safe intercept navigation in the presence of multiple vehicles

  • 248 Accesses

  • 1 Citations


This paper studies a safe intercept navigation which accounts for the uncertainty of other vehicles’ trajectories, avoids collisions and any other positions in which vehicle safety is compromised. Since the number of vehicles can vary with time, it is important that the navigation strategy can quickly adjust to the current number of vehicles, i.e, that it scales well with the number of vehicles. The scalable strategy is based on a stochastic optimal control problem formulation of safe navigation in the presence of a single vehicle, denoted as the one-on-one vehicle problem. It is shown that safe navigation in the presence of multiple vehicles can be solved exactly as an auxiliary Markov decision problem. This allows us to approximate the solution based on the one-on-one vehicle optimal control solution and achieve scalable navigation. Our work is illustrated by a numerical example of safely navigating a vehicle in the presence of four other vehicles and by a robot experiment.

This is a preview of subscription content, log in to check access.

Access options

Buy single article

Instant unlimited access to the full article PDF.

US$ 39.95

Price includes VAT for USA

Subscribe to journal

Immediate online access to all issues from 2019. Subscription will auto renew annually.

US$ 99

This is the net price. Taxes to be calculated in checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12


  1. Aigner, M., & Fromme, M. (1984). A game of cops and robbers. Discrete Applied Mathematics, 8(1), 1–12.

  2. Alonso-Mora, J., Breitenmoser, A., Rufli, M., Beardsley, P., & Siegwart, R. (2013). Optimal reciprocal collision avoidance for multiple non-holonomic robots (pp. 203–216). Berlin, Heidelberg: Springer.

  3. Anderson, R., & Milutinović, D. (2011). A stochastic approach to dubins feedback control for target tracking. In 2011 IEEE/RSJ international conference on intelligent robots and systems (pp. 3917–3922).

  4. Anderson, R. P., & Milutinović, D. (2014). A stochastic approach to dubins vehicle tracking problems. IEEE Transactions on Automatic Control, 59(10), 2801–2806.

  5. Ardema, M. D., Heymann, M., & Rajan, N. (1985). Combat games. Journal of Optimization Theory and Applications, 46(4), 391–398.

  6. Eklund, J., Sprinkle, J., Kim, H., & Sastry, S. (2005). Implementing and testing a nonlinear model predictive tracking controller for aerial pursuit/evasion games on a fixed wing aircraft. In 2005 American control conference (ACC) (Vol. 3, pp. 1509–1514).

  7. Festa, A., & Vinter, R. B. (2016). Decomposition of differential games with multiple targets. Journal of Optimization Theory and Applications, 169, 849–875.

  8. Fleming, W. H., & Rishel, R. W. (1975). Deterministic and stochastic optimal control. New York: Springer.

  9. Gardiner, C. (2009). Stochastic methods: A handbook for the natural and social sciences. Berlin, Heidelberg: Springer.

  10. Getz, W. M., & Leitmann, G. (1979). Qualitative differential games with two targets. Journal of Mathematical Analysis and Applications, 68, 421–430.

  11. Getz, W. M., & Pachter, M. (1981). Capturability in a two-target “game of two cars”. Journal of Guidance and Control, 4(1), 15–22.

  12. Grimm, W., & Well, K. H. (1991). Modelling air combat as differential game recent approaches and future requirements. In R. P. Hämäläinen, & H. K. Ehtamo (Eds.), Differential games—Developments in modelling and computation. Lecture notes in control and information sciences (Vol. 156). Berlin, Heidelberg: Springer.

  13. Hashemi, A., Casbeer, D. W., & Milutinović, D. (2016). Scalable value approximation for multiple target tail-chase with collision avoidance. In 2016 IEEE 55th conference on decision and control (CDC) (pp. 2543–2548).

  14. Hoy, M., Matveev, A., & Savkin, A. (2015). Algorithms for collision-free navigation of mobile robots in complex cluttered environments: A survey. Robotica, 33(3), 463–497.

  15. Huang, H., Ding, J., Zhang, W., & Tomlin, C. J. (2015). Automation-assisted capture-the-flag: A differential game approach. IEEE Transactions on Control Systems Technology, 23(3), 1014–1028.

  16. Isaacs, R. (1965). Differential games. New York, NY: Wiley.

  17. Israelsen, B. W., Ahmed, N., Center, K., Green, R., & Bennett Jr., W. (2017). Adaptive simulation-based training of ai decision-makers using bayesian optimization. arxiv:1703.09310.

  18. Kushner, H. J., & Dupuis, P. (2001). Numerical methods for stochastic control problems in continuous time, stochastic modelling and applied probability (Vol. 24). New York, NY: Springer.

  19. Li, D., Cruz, J. B., & Schumacher, C. J. (2008). Stochastic multi-player pursuit-evasion differential games. International Journal of Robust and Nonlinear Control, 18(6), 218–247.

  20. McGrew, J. S., How, J. P., Williams, B., & Roy, N. (2010). Air-combat strategy using approximate dynamic programming. Journal of Guidance, Control, and Dynamics, 33(5), 1509–1514.

  21. Milutinović, D., Casbeer, D. W., Kingston, D., & Rasmussen, S. A. (2017). Stochastic approach to small uav feedback control for target tracking and blind spot avoidance. In Proceedings of the 1st IEEE conference on control technology and applications.

  22. Munishkin, A. A., Milutinović, D., & Casbeer, D. W. (2016). Stochastic optimal control navigation with the avoidance of unsafe configurations. In 2016 international conference on unmanned aircraft systems (ICUAS) (pp. 211–218).

  23. Panagou, D., Stipanović, D. M., & Voulgaris, P. G. (2016). Distributed coordination control for multi-robot networks using Lyapunov-like barrier functions. IEEE Transactions on Automatic Control, 61(3), 617–632.

  24. Powell, W. B. (2009). What you should know about approximate dynamic programming. Naval Research Logistics (NRL), 56(3), 239–249.

  25. Song, Q., & Yin, G. G. (2010). Convergence rates of Markov chain approximation methods for controlled diffusions with stopping. Journal of Systems Science and Complexity, 23(3), 600–621.

  26. Vidal, R., Shakernia, O., Kim, H. J., Shim, D. H., & Sastry, S. (2002). Probabilistic pursuit-evasion games: Theory, implementation, and experimental evaluation. IEEE Transactions on Robotics and Automation, 18(5), 662–669.

  27. Vieira, M. A. M., Govindan, R., & Sukhatme, G. S. (2009). Scalable and practical pursuit-evasion with networked robots. Intelligent Service Robotics, 2(4), 247.

  28. Virtanen, K., Karelahti, J., & Raivio, T. (2006). Modeling air combat by a moving horizon influence diagram game. Journal of Guidance, Control, and Dynamics, 29(5), 1509–1514.

  29. Wang, L., Ames, A. D., & Egerstedt, M. (2017). Safety barrier certificates for collisions-free multirobot systems. IEEE Transactions on Robotics, 33(3), 661–674.

  30. Yavin, Y. (1988). Stochastic two-target pursuit-evasion differential games in the plane. Journal of Optimization Theory and Applications, 56(3), 325–343.

  31. Yavin, Y., & Villers, R. D. (1988). Stochastic pursuit-evasion differential games in 3D. Journal of Optimization Theory and Applications, 56(3), 345–357.

Download references


Funding was provided by U.S. Department of Defense (Grant No. FA8650-15-D-2516).

Author information

Correspondence to Dejan Milutinović.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 2 (mp4 6697 KB)

Supplementary material 1 (mp4 1917 KB)

Supplementary material 2 (mp4 6697 KB)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Munishkin, A.A., Hashemi, A., Casbeer, D.W. et al. Scalable Markov chain approximation for a safe intercept navigation in the presence of multiple vehicles. Auton Robot 43, 575–588 (2019).

Download citation


  • Autonomous navigation
  • Dubins vehicles
  • Stochastic optimal control