Optimal Control for Diffusion Processes

Part of the Probability Theory and Stochastic Modelling book series (PTSM, volume 72)


This chapter deals with completely observable stochastic control problems for diffusion processes, described by SDEs. The decision maker chooses an optimal decision among all possible ones to achieve the goal. Namely, for a control process, its response evolves according to a (controlled) SDE and the payoff on a finite time interval is given. The controller wants to minimize (or maximize) the payoff by choosing an appropriate control process from among all possible ones. Here we consider three types of control processes:
  1. 1.

    \((\mathcal{F}_{t})\)-progressively measurable processes.

  2. 2.

    Brownian-adapted processes.

  3. 3.

    Feedback controls.


In order to analyze the problems, we mainly use the dynamic programming principle (DPP) for the value function.The reminder of this chapter is organized as follows. Section 2.1 presents the formulation of control problems and basic properties of value functions, as preliminaries for later sections. Section 2.2 focuses on DPP. Although DPP is known as a two stage optimization method, we will formulate DPP by using a semigroup and characterize the value function via the semigroup. In Sect. 2.3, we deal with verification theorems, which give recipes for finding optimal Markovian policies. Section 2.4 considers a class of Merton-type optimal investment models, as an application of previous results.


Optimal Markovian Policies Dynamic Programming Principle (DPP) Verification Theorem Stochastic Control Problem Stage Optimization Method 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. [A03]
    R.A. Adams, Sobolev Spaces, 2nd edn. (Academic, Amsterdam/Boston, 2003)zbMATHGoogle Scholar
  2. [BC09]
    A. Bain, D. Crisan, Fundamentals of Stochastic Filtering (Springer, New York/ London, 2009)CrossRefzbMATHGoogle Scholar
  3. [Be52]
    R. Bellman, On the theory of dynamic programming. Proc. Nat. Sci. U.S.A. 38, 716–719 (1952)CrossRefzbMATHGoogle Scholar
  4. [Be57]
    R. Bellman, Dynamic Programming (Princeton University Press, Princeton, 1957)zbMATHGoogle Scholar
  5. [Be75]
    V.E. Benes, Composition and invariance methods for solvinig some stochastic control problems. Adv. Appl. Prob. 7, 299–329 (1975)CrossRefzbMATHMathSciNetGoogle Scholar
  6. [BSW80]
    V.E. Venes, L.A. Shepp, H.S. Witsenhaussen, Some solvable stochastic control problems. Stochastics 4, 39–83 (1980)CrossRefMathSciNetGoogle Scholar
  7. [Be92]
    A. Bensoussan, Stochastic Control of Partially Observable Systems (Cambridge University Press, Cambridge/New York, 1992)CrossRefzbMATHGoogle Scholar
  8. [BN90]
    A. Bensoussan, M. Nisio, Nonlinear semigroup arising in the control of diffusions with partial observation. Stoch. Stoch. Rep. 30, 1–45 (1990)CrossRefzbMATHMathSciNetGoogle Scholar
  9. [BP99]
    T.R. Bieleckii, S.R. Pliska, Risk sensitive dynamic asset management. Appl. Math. Optim. 39, 337–366 (1999)CrossRefMathSciNetGoogle Scholar
  10. [BS73]
    F. Black, M. Scholes, The pricing of options and corporate liabilities. J. Polit. Econ. 81, 637–659 (1973)CrossRefGoogle Scholar
  11. [BM07]
    R. Buckdahn, J. Ma, Pathwise stochastic control problems and stochastic HJB equations. SIAM J. Control Optim. 45, 2224–2256 (2007)CrossRefzbMATHMathSciNetGoogle Scholar
  12. [Br06]
    S. Brendle, Portfolio selection under incomplete information. Stoch. Proc. Appl. 116, 701–723 (2006)CrossRefzbMATHMathSciNetGoogle Scholar
  13. [CPY09]
    M.H. Chsng, T. Pang, J. Yong, Optimal stopping problem for stochastic differential equations with random cofficients. SIAM J. Control Optim. 48, 941–971 (2009)CrossRefMathSciNetGoogle Scholar
  14. [CI90]
    M.G. Crandall, H. Ishii, The maximum principle for semicontinuous functions. Differ. Integral Equ. 3, 1001–1014 (1990)zbMATHMathSciNetGoogle Scholar
  15. [CIL92]
    M.G. Crandall, H. Ishii, P.L. Lions, A user’s guide to viscosity solutions. Bull. AMS NS 27, 1–67 (1992)CrossRefzbMATHMathSciNetGoogle Scholar
  16. [DaPZ92]
    G. Da Prato, J. Zabczyk, Stochastic Equations in Infinite Dimenmensions (Cambridge University Press, Cambridge, 1992)CrossRefGoogle Scholar
  17. [ElKPQ97]
    N. El Karoui, S. Peng, M.C. Quenez, Backward stochastic differential equations in finance. Math. Finance 7, 1–71 (1997)CrossRefzbMATHMathSciNetGoogle Scholar
  18. [ElKQ95]
    N. El Karoui, M.C. Quenez, Dynamic programming and pricing of contingent claims in an incomplete market. SIAM. J. Control Optim. 33, 29–66 (1950)CrossRefGoogle Scholar
  19. [E83]
    L.C. Evans, Classical solutions of Hamilton-Jacobi-Bellman equation for uniformly elliptic operators. Trans. AMS 275, 245–255 (1983)CrossRefGoogle Scholar
  20. [E98]
    L.C. Evans, Partial Differential Equations. GSM19 (AMS, Providence, 1998)Google Scholar
  21. [FH11]
    W.H. Fleming, D. Hernandez-Hernandez, On the value of stochastic differential games. Commun. Stoch. Anal. 5, 341–351 (2011)MathSciNetGoogle Scholar
  22. [FKSh10]
    W.H. Fleming, H. Kaise, S.J. Sheu, Max-plus stochastic control and risk sensitivity. Appl. Math. Optim 62, 81–144 (2010)CrossRefzbMATHMathSciNetGoogle Scholar
  23. [FMcE95]
    W.H. Fleming, W.H. McEneaney, Risk sensitive control on infinite time horizon. SIAM J. Control Optim. 33, 1881–1915 (1995)CrossRefzbMATHMathSciNetGoogle Scholar
  24. [FR75]
    W.H. Fleming, R.W. Rishel, Deterministic and Stochastic Optimal Control (Springer, Berlin/New York, 1975)CrossRefzbMATHGoogle Scholar
  25. [FSh99]
    W.H. Fleming, S.J. Sheu, Optimal long term growth rate of expected utility of wealth. Ann. Appl. Prob. 9, 871–903 (1999)CrossRefzbMATHMathSciNetGoogle Scholar
  26. [FSh00]
    W.H. Fleming, S.J. Sheu, Risk sensitive control and an optimal investment model. Math. Finance 10, 197–213 (2000)CrossRefzbMATHMathSciNetGoogle Scholar
  27. [FSh02]
    W.H. Fleming, S.J. Sheu, Risk sensitive control and an optimal investment model II. Ann. Appl. Prob. 12, 730–767 (2002)CrossRefzbMATHMathSciNetGoogle Scholar
  28. [FS06]
    W.H. Fleming, H.M. Soner, Controlled MarkovProcesses and Viscosity Solutions, 2nd edn. (Springer, New York 2006)Google Scholar
  29. [FSo89]
    W.H. Fleming, P.E. Souganidis, On the existence of value function of two-plsyer, zero-sum stochastic differential games. Indiana Math. J. 38, 293–314 (1989)CrossRefzbMATHMathSciNetGoogle Scholar
  30. [FKK72]
    M. Fujisaki, G. Kallianpur, H. Kunita, Stochastic differential equations for the nonlinear filtering problem. Osaka J. Math. 9, 19–40 (1972)zbMATHMathSciNetGoogle Scholar
  31. [GŚ00]
    F. Gozzi, A. Świech, Hamilton-Jacobi-bellman equations for the optimal control of the Duncan-Mortensen-Zakai equation. J. Funct. Analy. 172, 466–510 (2000)CrossRefzbMATHGoogle Scholar
  32. [HP81]
    J.M. Harrison, S.R. Pliska, Martingales and stochastic integrals in the theory of continuous ytading. Stoch. Proc. Appl. 11, 215–260 (1981)CrossRefzbMATHMathSciNetGoogle Scholar
  33. [HP83]
    J.M. Harrison, S.R. Pliska, Stochastic calculus model of continuous trading; complete markets. Stoch. Proc. Appl. 15, 313–316 (1983)CrossRefzbMATHMathSciNetGoogle Scholar
  34. [HNSh10]
    H. Hata, H. Nagai, S.I. Sheu, Asymptotics of the probability minimizing a down-side risk. Ann. Appl. Prob. 20, 52–89 (2010)CrossRefzbMATHMathSciNetGoogle Scholar
  35. [HS10]
    H. Hata, J. Sekine, Explicit solution to a certain nonELQG risk-sensitive stochastic control problem. Appl. Math. Optim. 62, 341–380 (2010)CrossRefzbMATHMathSciNetGoogle Scholar
  36. [IW81]
    N. Ikeda, S. Watanabe, Stochastic Differential Equations and Diffusion Processes (North Holland, Amsterdam/New York, 1981)zbMATHGoogle Scholar
  37. [Is92]
    H. Ishii, Viscosity solutions for a class of Hamilton-Jacobi equations in Hilbert spaces. J. Funct. Anal. 105, 301–341 (1992)CrossRefzbMATHMathSciNetGoogle Scholar
  38. [I42]
    K. Itô, Differential equations determining Markov processes. Zenkoku Shijo Sugaku Danwakai 244, 1352–1400 (1942). (In Japanese)Google Scholar
  39. [I51]
    K. Itô, On Stochastic Differential Equations. Memoirs of the American Mathematical Society, vol. 4 (AMS, New York City, 1951)Google Scholar
  40. [JLL90]
    P. Jaillet, D. Lamberton, B. Lapeyer, Variational inequalities and the pricing of American options. Acta Appl. Math. 21, 263–289 (1990)CrossRefzbMATHMathSciNetGoogle Scholar
  41. [KS91]
    I. Karatzas, S.E. Sheve, Brownian Motion and Stochastic Calculus, 2nd edn. (Springer, New York, 1991)zbMATHGoogle Scholar
  42. [KS98]
    I. Karatzas, S.E. Sheve, Methods of Mathematical Finance (Springer, New York, 1998)CrossRefzbMATHGoogle Scholar
  43. [Ko04]
    S. Koike, A Biginner’s Guide to the Theory of Viscisity Solutions. MSJ Memoirs, vol. 13 (JMS, Tokyo, 2004)Google Scholar
  44. [KN02]
    K. Kuroda, H. Nagai, Risk sensitive portfolio optimization and infinite time horizon. Stoch. Stoch. Rep. 73, 309–331 (2002)CrossRefzbMATHMathSciNetGoogle Scholar
  45. [Kr09]
    N.V. Krylov, Controlled Diffusion Processes, 2nd edn. (Springer, Berlin, 2009)zbMATHGoogle Scholar
  46. [Ku67]
    H.J. Kushner, Dynamical equations for optimal nonlinear filtering. J. Differ. Equ. 3, 179–190 (1967)CrossRefzbMATHMathSciNetGoogle Scholar
  47. [La83]
    S. Lang, Real Analysis, 2nd edn. (Addison-Wesley, New York, 1983)zbMATHGoogle Scholar
  48. [L83]
    P.L. Lions, Optimal control of diffusion processes and Hamilton-Jacobi-Bellman equations I. J. Commun. PDE. 8, 1101–1134 (1983)CrossRefzbMATHGoogle Scholar
  49. [L88]
    P.L. Lions, Viscosity solutions of fully nonlinear second-order equations and optimal stochastic control in infinite dimensions. Part I. The case of bounded stochastic evolution. Acta Math. 161, 243–278 (1988)zbMATHGoogle Scholar
  50. [L89]
    P.L. Lions, Viscosity solutions of fully nonlinear second-order equations and optimal stochastic control in infinite dimensions. Part II. Optimal control of Zakai equation, in Stochastic Partial Differential Equations and Applications II, ed. by G. Da Prato, L. Tubaro. Lecture Notes in Mathematics, vol. 1390 (Springer, Berlin/Heidelberg, 1989), pp. 147–170. Part III. Uniqueness of viscosity solutions for general second order equations. J. Funct. Anal. 86, 1–18 (1989)Google Scholar
  51. [LN83]
    P.L. Lions, M. Nisio, A uniqueness result for the semigroup associated with HJB equations. Proc. Jpn. Acad. 58, 273–276 (1983)CrossRefMathSciNetGoogle Scholar
  52. [LS01]
    R.S. Liptser, A.N. Shiryayev, Statistics of Random Processes I, II, 2nd edn. (Springer, New York/Berlin, 2001)CrossRefGoogle Scholar
  53. [Ma00]
    C. Martini, American option prices as unique viscosity solutions to degenerate HJB equations, Rapport de rech, INRIA, 2000Google Scholar
  54. [Me71]
    R.C. Merton, Optimal consumption and portfolio rules in continuous time. J. Econ. Theory 3, 373–413 (1971)CrossRefzbMATHMathSciNetGoogle Scholar
  55. [Me73]
    R.C. Merton, Theory of rational option pricing. Bell J. Econ. Manag. Sci. 4, 141–183 (1973)CrossRefMathSciNetGoogle Scholar
  56. [M88]
    M. Metivier, Stochastic Partial Differential Equations in Infinite Dimensional Spaces (Scuola Superiore, Pisa, 1988)zbMATHGoogle Scholar
  57. [My66]
    P.A. Meyer, Probability and Potentials (Blaisdell, Waltham, 1966)zbMATHGoogle Scholar
  58. [Mo10]
    H. Morimoto, Stochastic Control and Mathematical Modeling, Applications in Economics (Cambridge University Press, Cambridge/New York, 2010)CrossRefzbMATHGoogle Scholar
  59. [Na03]
    H. Nagai, Optimal strategies for risk sensitive portfolio optimization problems for general factor models. SIAM J. Control Optim. 41, 1779–1800 (2003)CrossRefzbMATHMathSciNetGoogle Scholar
  60. [N76]
    M. Nisio, On a nonlinear semigroup attached to stochastic optimal control. Pull. RIMS 12, 513–537 (1976)CrossRefzbMATHMathSciNetGoogle Scholar
  61. [N78]
    M. Nisio, On nonlinear semigroups for Markov processes associated with optimal stopping. Appl. Math. Optim. 4, 143–169 (1978)CrossRefzbMATHMathSciNetGoogle Scholar
  62. [N81]
    M. Nisio, Lecture on Stochastic Control Theory. ISI Lecture Notes, vol. 9 (McMillan India, Delhi 1981)Google Scholar
  63. [N88]
    M. Nisio, Stochastic differential games and viscosity solutions of Isaacs equations. Nagoya Math. J. 110, 163–184 (1988)zbMATHMathSciNetGoogle Scholar
  64. [P79]
    E. Pardoux, Stochastic partial differential equations and filtering of diffusion processes. Stochastics 3, 127–167 (1979)CrossRefzbMATHMathSciNetGoogle Scholar
  65. [P93]
    E. Pardoux, Stochastic partial differential equations, a review. Bull. Sc. Math. 117, 29–47 (1993)zbMATHMathSciNetGoogle Scholar
  66. [PP90]
    E. Pardoux, S. Peng, Adapted solution of backward stoshastic equation. Syst. Control Lett. 14, 55–61 (1990)CrossRefzbMATHMathSciNetGoogle Scholar
  67. [Pe92]
    S. Peng, Stochastic Hamilton-Jacobi-Bellman equations. SIAM J. Control Optim. 30, 284–304 (1992)CrossRefzbMATHMathSciNetGoogle Scholar
  68. [Ph02]
    H. Pham, Smooth solutions to optimal investment models with stochastic volatilities and portfolio constraints. Appl. Math. Optim. 46, 55–78 (2002)CrossRefzbMATHMathSciNetGoogle Scholar
  69. [R90]
    B.L. Rozovskii, Stochastic Evolution Systems (Kluwer Academic, Dordrecht/Boston, 1990)CrossRefzbMATHGoogle Scholar
  70. [S08]
    A.N. Shiryayev, Optimal Stopping Rules, 2nd edn. (Springer, Berlin/Heidelberg 2008)Google Scholar
  71. [St11]
    L. Stettner, Penalty method for finite horizon stopping problems. SIAM J. Control Optim. 49, 1078–1099 (2011)CrossRefzbMATHMathSciNetGoogle Scholar
  72. [SV79]
    D.V. Strook, S.R.S. Varadhan, Multidimensional Diffusion Processes (Springer, Berlin/New York, 1979)Google Scholar
  73. [W71]
    J.C. Willems, Least squares stationary optimal control and the algebraic Ricatti equation. IEEE. Trans. Auto. Control 16, 621–635 (1971)CrossRefMathSciNetGoogle Scholar
  74. [Wo68]
    W.M. Wonham, on a matrix Ricatti equation of stochastic control. SIAM J. Control Optim. 6, 681–697 (1968)Google Scholar
  75. [Y80]
    K. Yosida, Functional Analysis, 6th edn. (Springer, Berlin/New York, 1980)CrossRefzbMATHGoogle Scholar
  76. [YZ99]
    J. Yong, X.Y. Zhou, Stochastic Controls, Hamiltonian Systems and HJB Equations (Springer, New York, 1999)zbMATHGoogle Scholar

Copyright information

© Springer Japan 2015

Authors and Affiliations

  1. 1.Kobe UniversityKobeJapan
  2. 2.Osaka Electro–Communication UniversityOsakaJapan

Personalised recommendations