A Continuous-Valued Learning Controller for the global optimization of Stochastic Control Systems

Abstract

The optimization of stochastic control systems has received increased attention in recent years [1, 2, 3, 4]. The stochastic nature of a control situation can arise from a variety of sources such as measurement noise, parameter drift, or other, possibly external disturbances. Such random properties represent a description of inaccurately known disturbances, signals, or system changes. If there exists much a priori information or assumptions concerning plant dynamics and/or statistical properties, then a stochastic controller selecting control policies on the basis of a state estimation procedure may be appropriate. With known statistical properties, the estimation is amenable to a Kaiman filter or a Bayesian approach. However, the lack of sufficient a priori assumptions concerning the stochastic plant dynamics requires a learning controller to interact with the stochastic plant to gain information for improving performance. Several algorithms for the learning controller for seeking optimal control policies have been considered elsewhere [5, 6, 7]. One such algorithm which has received considerable attention is stochastic approximation; it is a stochastic form of a more general class of “gradient” or hill-climbing search algorithms. Such algorithms, however, tend to have local optimization properties. Hence, although often providing rapid convergence, such an algorithm can “miss” an extremum while seeking a local minimum or maximum.