Indoor Pursuit-Evasion with Hybrid Hierarchical Partially Observable Markov Decision Processes for Multi-robot Systems

  • Sha YiEmail author
  • Changjoo Nam
  • Katia Sycara
Conference paper
Part of the Springer Proceedings in Advanced Robotics book series (SPAR, volume 9)


In this paper, we examine a pursuit-evasion problem where more than one pursuer may search for one evader in indoor environments. Partially Observable Markov Decision Processes (POMDPs) provide a framework to model the uncertainty arisen from the unknown location of the evader. However, the approach is intractable even with a single pursuer and an evader. Therefore, we propose a Hybrid Hierarchical POMDP structure for improved scalability and efficiency. The structure consists of (i) the base MDPs for the cases where the evader is visible to the pursuers, (ii) the abstract POMDPs for the evader states that are not directly observable, and (iii) the transition states bridging between the base MDPs and abstract POMDPs. This hybrid approach significantly reduces the number of states expanded in the policy tree to solve the problem by abstracting environment structures. Experimental results show that our method expands only 5% of nodes generated from a standard POMDP solution.


Pursuit and evasion Multi-robot systems Markov decision processes 


  1. 1.
    Arai, S., Sycara, K., Payne, T.R.: Experience-Based Reinforcement Learning to Acquire Effective Behavior in a Multi-agent Domain, pp. 125–135. Springer (2000)Google Scholar
  2. 2.
    Bellman, R.: Dynamic Programming. Courier Corporation (2013)Google Scholar
  3. 3.
    Eddy, W.F.: A new convex hull algorithm for planar sets. ACM Trans. Math. Softw. (TOMS) 3(4), 398–403 (1977)CrossRefGoogle Scholar
  4. 4.
    Gopalan, N., des Jardins, M., Littman, M.L., MacGlashan, J., Squire, S., Tellex, S., Winder, J., Wong, L.L.: Planning with Abstract Markov Decision Processes (2017)Google Scholar
  5. 5.
    Guibas, L.J., Latombe, J.C., LaValle, S.M., Lin, D., Motwani, R.: A visibility-based pursuit-evasion problem. Int. J. Comput. Geom. Appl. 9(04), 471–493 (1999)MathSciNetCrossRefGoogle Scholar
  6. 6.
    Hauskrecht, M.: Value-function approximations for partially observable markov decision processes. J. Artif. Intell. Res. 13, 33–94 (2000)MathSciNetCrossRefGoogle Scholar
  7. 7.
    Hollinger, G., Kehagias, A., Singh, S.: Probabilistic strategies for pursuit in cluttered environments with multiple robots. In: IEEE International Conference on Robotics and Automation, pp. 3870–3876. IEEE (2007)Google Scholar
  8. 8.
    Hollinger, G., Singh, S., Djugash, J., Kehagias, A.: Efficient multi-robot search for a moving target. Int. J. Robot. Res. 28(2), 201–219 (2009)CrossRefGoogle Scholar
  9. 9.
    Isler, V., Sun, D., Sastry, S.: Roadmap based pursuit-evasion and collision avoidance. Robot. Sci. Syst. 1, 257–264 (2005)Google Scholar
  10. 10.
    Ong, S.C., Png, S.W., Hsu, D., Lee, W.S.: Planning under uncertainty for robotic tasks with mixed observability. Int. J. Robot. Res. 29(8), 1053–1068 (2010)CrossRefGoogle Scholar
  11. 11.
    Papadimitriou, C.H., Tsitsiklis, J.N.: The complexity of markov decision processes. Math. Oper. Res. 12(3), 441–450 (1987)MathSciNetCrossRefGoogle Scholar
  12. 12.
    Pynadath, D.V., Tambe, M.: The communicative multiagent team decision problem: analyzing teamwork theories and models. J. Artif. Intell. Res. 16, 389–423 (2002)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Robotics InstituteCarnegie Mellon UniversityPittsburghUSA

Personalised recommendations