A Hamilton-Jacobi-Bellman Approach for the Numerical Computation of Probabilistic State Constrained Reachable Sets

  • Mohamed Assellaou
  • Athena PicarelliEmail author
Part of the Springer INdAM Series book series (SINDAMS, volume 29)


Aim of this work is to characterise and compute the set of initial conditions for a system of controlled diffusion processes which allow to reach a terminal target satisfying pointwise state constraints with a given probability of success. Defining a suitable auxiliary optimal control problem, the characterization of this set is related to the solution of a particular Hamilton-Jacobi-Bellman equation. A semi-Lagrangian numerical scheme is defined and its convergence to the unique viscosity solution of the equation is proved. The validity of the proposed approach is then tested on some numerical examples.


Viscosity solutions Reachable set Discontinuous cost functions Neumann boundary conditions 



The authors are sincerely grateful to Olivier Bokanowski and Hasnaa Zidani for their guidance at the early stage of this paper.


  1. 1.
    Abate, A., Amin, S., Prandini, M., Lygeros, J., Sastry, S.: Computational approaches to reachability analysis of stochastic hybrid systems. In: Hybrid Systems. Lecture Notes in Computer Science, vol. 4416(1), pp. 4–17 (2007)MathSciNetzbMATHGoogle Scholar
  2. 2.
    Abate, A., Prandini, M., Lygeros, J., Sastry, S.: Probabilistic reachability and safety for controlled discrete time stochastic hybrid systems. Automatica 44, 2724–2734 (2008)MathSciNetCrossRefGoogle Scholar
  3. 3.
    Althoff, M., Stursberg, O., Buss, M.: Safety assessment of autonomous cars using verification techniques. In: 2007 American Control Conference, pp. 4154–4159 (2007)Google Scholar
  4. 4.
    Althoff, M., Stursberg, O., Buss, M.: Safety assessment for stochastic linear systems using enclosing hulls of probability density functions. In: 2009 European Control Conference (ECC), pp. 625–630 (2009)Google Scholar
  5. 5.
    Althoff, M., Stursberg, O., Buss, M.: Safety assessment for stochastic linear systems using enclosing hulls of probability density functions. In: European Control Conference (ECC), pp. 625–630. IEEE (2009)Google Scholar
  6. 6.
    Assellaou, M., Bokanowski, O., Zidani, H.: Error estimates for second order hamilton-jacobi-bellman equations. approximation of probabilistic reachable sets. Discrete Contin. Dynam. Syst. Ser. A 35(9), 3933–3964 (2015)Google Scholar
  7. 7.
    Barles, G., Souganidis, P.E.: Convergence of approximation schemes for fully nonlinear second order equations. Asymptot. Anal. 4, 271–283 (1991)MathSciNetzbMATHGoogle Scholar
  8. 8.
    Barron, E.N.: The Bellman equation for control of the running max of a diffusion and applications to lookback options. Appl. Anal. 48, 205–222 (1993)MathSciNetCrossRefGoogle Scholar
  9. 9.
    Bokanowski, O., Forcadel, N., Zidani, H.: Reachability and minimal times for state constrained nonlinear problems without any controllability assumption. SIAM J. Control Optim. 48(7), 4292–4316 (2010)MathSciNetCrossRefGoogle Scholar
  10. 10.
    Bokanowski, O., Picarelli, A., Zidani, H.: Dynamic programming and error estimates for stochastic control problems with maximum cost. Appl. Math. Optim. 71(1), 125–163 (2015)MathSciNetCrossRefGoogle Scholar
  11. 11.
    Bouchard, B., Touzi, N.: Weak dynamic programming principle for viscosity solutions. SIAM J. Control Optim. 49(3), 948–962 (2011)MathSciNetCrossRefGoogle Scholar
  12. 12.
    Bouchard, B., Elie, R., Touzi, N.: Stochastic target problems with controlled loss. SIAM J. Control Optim. 48(5), 3123–3150 (2009)MathSciNetCrossRefGoogle Scholar
  13. 13.
    Camilli, F., Falcone, M.: An approximation scheme for the optimal control of diffusion processes. RAIRO Modél. Math. Anal. Numér. 29(1), 97–122 (1995)MathSciNetCrossRefGoogle Scholar
  14. 14.
    Crandall, M.G., Ishii, H., Lions, P.L.: User’s guide to viscosity solutions of second order partial differential equations. Bull. Am. Math. Soc. 27(1), 1–67 (1992)MathSciNetCrossRefGoogle Scholar
  15. 15.
    Debrabant, K., Jakobsen, E.R.: Semi-Lagrangian schemes for linear and fully non-linear diffusion equations. Math. Comp. 82(283), 1433–1462 (2012)MathSciNetCrossRefGoogle Scholar
  16. 16.
    Falcone, M., Giorgi, T., Loreti, P.: Level sets of viscosity solutions: some applications to fronts and rendez-vous problems. SIAM J. Appl. Math. 54, 1335–1354 (1994)MathSciNetCrossRefGoogle Scholar
  17. 17.
    Fleming, W.H., Soner, H.M.: Controlled Markov Processes and Viscosity Solutions, 2nd edn. Springer, New York (2006)zbMATHGoogle Scholar
  18. 18.
    Föllmer, H., Leukert, P.: Quantile hedging. SIAM J. Comput. Phys. 3(3), 251–273 (1999)MathSciNetzbMATHGoogle Scholar
  19. 19.
    Grüne, L., Picarelli, A.: Zubov’s method for controlled diffusions with state constraints. Nonlinear Differ. Equ. Appl. 22(6), 1765–1799 (2015)MathSciNetCrossRefGoogle Scholar
  20. 20.
    Kloeden, P.E., Platem, E.: Numerical Solution of Stochastic Differential Equations. Springer, Berlin/New York (1992)CrossRefGoogle Scholar
  21. 21.
    Kröner, A., Picarelli, A., Zidani, Z.: Infinite horizon stochastic optimal control problems with running maximum cost. SIAM J. Control Optim. 56(5), 3296–3319 (2018)MathSciNetCrossRefGoogle Scholar
  22. 22.
    Margellos, K., Lygeros, J.: Hamilton-Jacobi formulation for Reach-avoid differential games. IEEE Trans. Autom. Control 56, 1849–1861 (2011)MathSciNetCrossRefGoogle Scholar
  23. 23.
    Menaldi, J.L.: Some estimates for finite difference approximations. SIAM J. Control Optim. 27, 579–607 (1989)MathSciNetCrossRefGoogle Scholar
  24. 24.
    Mitchell, I., Bayen, A., Tomlin, C.: A time-dependent Hamilton-Jacobi formulation of reachable sets for continuous dynamic games. IEEE Trans. Autom. Control 50, 947–957 (2005)MathSciNetCrossRefGoogle Scholar
  25. 25.
    Osher, S., Sethian, A.J.: Fronts propagating with curvature dependent speed: algorithms on Hamilton-Jacobi formulations. J. Comp. Phys. 79, 12–49 (1988)MathSciNetCrossRefGoogle Scholar
  26. 26.
    Soner, H., Touzi, N.: A stochastic representation for level set equations. Commun. Partial Differ. Equ. 27(9–10), 2031–2053 (2002)MathSciNetCrossRefGoogle Scholar
  27. 27.
    Yong, J., Zhou, X.Y.: Stochastic Controls. Applications of Mathematics (New York), vol. 43. Springer, New York (1999). Hamiltonian Systems and HJB EquationsGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.ENSTA ParisTechPalaiseau CedexFrance
  2. 2.Department of Economical SciencesUniversity of VeronaVeronaItaly

Personalised recommendations