Skip to main content
SpringerLink
Log in

Optimal Control of Diffusion Processes

  • Chapter
  • First Online:
Relative Optimization of Continuous-Time and Continuous-State Stochastic Systems

Part of the book series: Communications and Control Engineering ((CCE))

  • 347 Accesses

Abstract

In this chapter, we study optimization problems with diffusion processes for long-run average, finite-horizon, optimal stopping, and singular control. The value function for finite-horizon problems, or the potential function for long-run average, can be smooth or semi-smooth (both one-sided first-order derivatives exist, but not equal). Explicit optimality conditions are derived at both smooth and semi-smooth points. This extends the famous Hamilton-Jacobi-Bellman (HJB) equations from smooth value functions to semi-smooth value functions, which cover the degenerate diffusion processes. Viscosity solution is not used. The performance-difference formula is based on the Ito-Tanaka formula for semi-smooth functions, which involves local time in \([t, t+ dt]\) with a mean of the order of \(\sqrt{dt}\). We also show that under some conditions, the semi-smoothness of the value (or potential) functions can simply be ignored.

The significant problems we face cannot be solved at the same level of thinking we were at when we created them [1].

Albert Einstein

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    This can be replaced by a weaker condition, the significant variance (SV) condition: \(| \frac{\mu (x)}{\sigma (x) }|<K< \infty \), \(x \in {\mathscr {S}}\), see [28].

  2. 2.

    When \(\sigma ' (x)=0\), the local time \(L_x^{X'} (t) =0\), and the difference formula still holds, with the second term on its right-hand side disappeared. However, \(X'(t)\) is degenerate at x, and it behaves differently from a non-degenerate point; in particular, in a long term, \(X'(t)\) stays in only one side of x. Some special consideration is needed; see Chap. 4 for details.

  3. 3.

    We expect that the results, i.e., (3.72) and what follows, hold if g(x) has a countable sequence of semi-smooth points as well; however, the proof requires the theory of local time at countable many semi-smooth points, which is beyond the scope of this paper.

  4. 4.

    By the Girsanov theorem, with the Radon–Nikodym derivative ((B.19) in Appendix B), any diffusion process in a finite period [0, T] can be transformed to a Brownian motion under another measure, and the system becomes the same as in Example 3.12. But this technique may not work well for infinite horizon processes because the integration in (B.19) may be infinite. In this example, we adopt another approach.

  5. 5.

    This problem is in fact an optimization problem with the following constraint on the actions taken at different states x: \(u (x) \equiv \mu \). The optimality is clearly shown by the performance-difference formula (3.95).

  6. 6.

    This can also be obtained by the probability density function of X(t) (A.37).

  7. 7.

    In the literature of stochastic control, it is shown that the value function is the viscosity solution to the HJB equation, or the first equation in (3.150). In other words, the HJB equation holds at the smooth points of the value function, and viscosity property (see Problem 3.18) holds at the non-smooth points of the value function [9, 12, 22, 37].

  8. 8.

    We may verify that \(\widehat{\eta }(x) := \eta ^{\widehat{u}}\) satisfying the two conditions in (3.150) is a viscosity solution to its first equation, as specified in Definition 9.1 of [5].

  9. 9.

    A mathematical proof of the existence and uniqueness of the solution to (3.150) remains for future research, also see the discussion on “smooth fit” in the next subsection.

  10. 10.

    A European call option discussed in Sect. 3.1.4 has a fixed maturity time T.

References

  1. Maxwell JC (2002) The 17 essential qualities of a team player: becoming the kind of person every team wants. Thomas Nelson, Nashville

    Google Scholar 

  2. Ikeda N, Watanabe A (1989) Stochastic differential equations and diffusion processes. North-Holland Publishing Company, Amsterdam

    MATH  Google Scholar 

  3. Karatzas I, Shreve SE (1991) Brownian motion and stochastic calculus, 2nd edn. Springer, Berlin

    MATH  Google Scholar 

  4. Klebaner FC (2005) Introduction to stochastic calculus with applications, 2nd edn. Imperial College Press, London

    Book  MATH  Google Scholar 

  5. Øksendal B (2003) Stochastic differential equations: an introduction with applications, 6th edn. Springer, Berlin

    Book  MATH  Google Scholar 

  6. Protter PE (2000) Stochastic integrations and differential equations, 2nd edn. Springer, Berlin

    Google Scholar 

  7. Borkar VS (1989) Optimal control of diffusion processes, vol 203. Pitman research notes in mathematics series, Longman Scientific and Technical, Harlow

    MATH  Google Scholar 

  8. Brockett R (2009) Stochastic control. Harvard University, Cambridge, Lecture notes

    Google Scholar 

  9. Fleming WH, Soner HM (2006) Controlled markov processes and viscosity solutions, 2nd edn. Springer, Berlin

    MATH  Google Scholar 

  10. Kushner HJ (1977) Probability methods for approximations in stochastic control and for elliptic equations. Academic Press, New York

    MATH  Google Scholar 

  11. Taksar MI (2008) Diffusion optimization models in insurance and finance. University of Texas, Texas, Lecture notes

    Google Scholar 

  12. Yong J, Zhou XY (1999) Stochastic controls - Hamilton systems and HJB equations. Springer, Berlin

    MATH  Google Scholar 

  13. Revuz D, Yor M (1991) Continuous martingales and Brownian motion. Springer, Berlin

    Book  MATH  Google Scholar 

  14. Karatzas I, Shreve SE (1998) Methods of mathematical finance. Springer, Berlin

    Book  MATH  Google Scholar 

  15. Akian M, Sulem A, Taksar M (2001) Dynamic optimizaton of long-term growth rate for a portfolio with transaction costs and logarithmis utility. Math Financ 11(2):153–188

    Article  MATH  Google Scholar 

  16. Cao XR, Wan XW (2017) Sensitivity analysis of nonlinear behavior with distorted probability. Math Financ 27:115–150

    Article  MathSciNet  MATH  Google Scholar 

  17. Jacka SD (1991) Optimal stopping and the American put. Math Financ 1(2):1–14

    Article  MATH  Google Scholar 

  18. Jin H, Zhou XY (2008) Behavioral portfolio selection in continuous time. Math Financ 18:385–426

    Article  MathSciNet  MATH  Google Scholar 

  19. Kushner HJ, Yin G (1997) Stochastic approximation algorithms and applications. Springer, New York

    Book  MATH  Google Scholar 

  20. Soner HM (2003) Stochastic optimal control in finance. Cattedra Galileiana, Scuola Normale, Pisa

    MATH  Google Scholar 

  21. Tamura T (2008) Maximization of the long-term grouth rate for a portfolio with fixed and proportional transaction costs. Adv Appl Probab 40:673–695

    Article  MATH  Google Scholar 

  22. Øksendal B, Sulem A (2007) Applied stochastic control of jump diffusions. Springer, Berlin

    Book  MATH  Google Scholar 

  23. Meyer PA (1976) Un cours sur les integrales stochstiques. Lect Notes Math 511:245–398

    Article  Google Scholar 

  24. Tanaka H (1963) Note on continuous additive functionals of the 1-dimensional Brownian path. Z Wahrscheinlichkeitstheorie 1:251–257

    Article  MathSciNet  MATH  Google Scholar 

  25. Wang AT (1977) Generalized Ito’s formula and additive functions of Brownian motion. Z Wahrscheinlichkeitstheorie verw. Gebiete 40:153–159

    Article  MATH  Google Scholar 

  26. Linetsky V (2005) On the transition densities of reflected diffusions. Adv Appl Probab 37:435–460

    Article  MathSciNet  MATH  Google Scholar 

  27. Skorokhod AV (1961) Stochastic equations for diffusions in a bounded region. Theory Probab Appl 6:264–274

    Article  Google Scholar 

  28. Cao XR (2017) Relative time and stochastic control with non-smooth features. IEEE Trans Autom Control 62:837–852

    Article  MathSciNet  MATH  Google Scholar 

  29. Bass RF (1984) Joint continuity and representations of additive functionals of d-dimensional Brownian motion. Stoch Proc Appl 17:211–227

    Article  MathSciNet  MATH  Google Scholar 

  30. Bass RF, Khoshnevisan D (1992) Local times on curves and uniform invariance principles. Probab Theory Relat Fields 92:465–492

    Article  MathSciNet  MATH  Google Scholar 

  31. Davis B (1998) Distribution of Brownian local time on curves. Bull Lond Math Soc 30(2):182–184

    Article  MathSciNet  Google Scholar 

  32. Peskir G (2005) A change-of-variable formula with local time on curves. J Theor Probab 18:499–535

    Article  MathSciNet  MATH  Google Scholar 

  33. Cao XR (2017) Optimality conditions for long-run average rewards with under selectivity and non-smooth features. IEEE Trans Autom Control 62:4318–4332

    Article  MATH  Google Scholar 

  34. Cao XR (2007) Stochastic learning and optimization - a sensitivity-based approach. Springer, Berlin

    Book  MATH  Google Scholar 

  35. Bryson AE, Ho YC (1969) Applied optimal control: optimization, estimation, and control. Blaisdell, Waltham

    Google Scholar 

  36. Cao XR (2017) Stochastic feedback control with one-dimensional degenerate diffusions and non-smooth value functions. IEEE Trans Autom Control 62:6136–6151

    Article  MATH  Google Scholar 

  37. Reikvam K (1998) Viscosity solutions of optimal stopping problems. Stoch Stoch Rep 62:285–301

    Article  MathSciNet  MATH  Google Scholar 

  38. Benes VE, Shepp LA, Witsenhausen HS (1980) Some soluble stochastic control problems. Stochastics 39–83

    Google Scholar 

  39. Andersen LN, Asmussen S, Glynn PW, Pihlsgård M (2015) Lévy processes with two-sided reflection, Lévy Matters V. Lect Notes Math 2149:67–182

    Article  MATH  Google Scholar 

  40. Andersen LN, Asmussen S (2011) Local time asymptotics for centered Lévy processes with two-sided reflection. Stoch Model 27:202–219

    Article  MATH  Google Scholar 

  41. Asmussen S, Pihlsgård M (2007) Loss rate for Lévy processes with two reflecting barriers. Math Oper Res 32:308–321

    Article  MathSciNet  MATH  Google Scholar 

  42. Forde M, Kumar R, Zhang H (2015) Large deviations for boundary local time of doubly reflected Brownian motion. Stat Probab Lett 96:262–268

    Article  MathSciNet  MATH  Google Scholar 

  43. Glynn PW, Wang RJ (2015) Central limit theorems and large deviations for additive functionals of reflecting diffusion processes. In: Dawson D, Kulik R, Ould Haye M, Szyszkowicz B, Zhao Y (eds) Fields communications series: asymptotic laws and methods in stochastics. Springer, New York

    Google Scholar 

  44. Pihlsgård M, Glynn PW (2013) On the dynamics of semi-martingales with two reflecting barriers. J Appl Probab 50:671–685

    Article  MathSciNet  MATH  Google Scholar 

  45. Kruk L, Lehoczky J, Ramanan K, Shreve S (2007) An explicit formula for the Skorokhod map on \([0, a]\). Ann Probab 35:1740–1768

    Article  MathSciNet  MATH  Google Scholar 

  46. Bensoussan A, Liu J, Yuan J (2010) Singular control and impulse control: a common approach. Discret Contin Dyn Syst, Ser B 13:27–57

    MathSciNet  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Automation, Shanghai Jiao Tong University, Shanghai, China

    Xi-Ren Cao

  2. Professor Emeritus, Department of Electrical and Computer Engineering, The Hong Kong University of Science and Technology, Kowloon, Hong Kong

    Xi-Ren Cao

Authors
  1. Xi-Ren Cao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xi-Ren Cao .

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Cao, XR. (2020). Optimal Control of Diffusion Processes. In: Relative Optimization of Continuous-Time and Continuous-State Stochastic Systems. Communications and Control Engineering. Springer, Cham. https://doi.org/10.1007/978-3-030-41846-5_3

Download citation

Publish with us

Policies and ethics

Search

Navigation