The Markov Chain Approximation Method: Introduction

Abstract

The main purpose of the book is the development of numerical methods for the solution of control or optimal control problems, or for the computation of functionals of the stochastic processes of interest, of the type described in Chapters 3, 7–9, and 12–15. It was shown to Chapter 3 that the cost or optimal cost functionals can be the (at least formal) solutions to certain nonlinear partial differential equations. It is tempting to try to solve for or approximate the various cost functions and optimal controls by dealing directly with the appropriate PDE’s, and numerically approximating their solutions. A basic impediment is that the PDE’s often have only a formal meaning, and standard methods of numerical analysis might not be usable to prove convergence of the numerical methods. For many problems of interest, one cannot even write down a partial differential equation. The Bellman equation might be replaced by a system of “variational inequalities,” or the proper form might not be known. Optimal stochastic control problems occur in an enormous variety of forms. As time goes on, we learn more about the analytical methods which can be used to describe and analyze the various optimal cost functions, but even then it seems that many important classes of problems are still not covered and new models appear which need even further analysis. The optimal stochastic control or stochastic modeling problem usually starts with a physical model, which guides the formulation of the precise stochastic process model to be used in the analysis. One would like numerical methods which are able to conveniently exploit the intuition contained in the physical model.