Advances in Information Systems Science pp 251-292 | Cite as

# Learning Control Systems

- 3 Citations
- 71 Downloads

## Abstract

In designing an optimal control system, if all the *a priori* information about the controlled process (plant-environment) is known and can be described deterministically, the optimal controller is usually designed by deterministic optimization techniques. If all or a part of the *a priori* information can only be described statistically—for example, in terms of probability distribution or density functions—then stochastic or statistical design techniques will be used. However, if the *a priori* information required is unknown or incompletely known, in general an optimal design cannot be achieved. Two different approaches have been taken to solve this class of problems. One approach is to design a controller based only upon the amount of information available. In that case the unknown information is either ignored or is assumed to take on some known values chosen according to the designer’s best guess. The second approach is to design a controller which is capable of estimating the unknown information during its operation and of determining an optimal control action on the basis of the estimated information. In the first case a rather conservative design criterion (for example, the minimax criterion) is often used; the systems designed are in general inefficient and suboptimal. In the second case, if the estimated information gradually approaches the true information as time proceeds, then the controller thus designed will approach to the optimal controller.

## Keywords

Stochastic Approximation Control Situation Switching Surface Reinforcement Algorithm Stochastic Automaton## Preview

Unable to display preview. Download preview PDF.

## References

- 1.N. J. Nilsson, “Learning Machines,“ McGraw-Hill Book Co., New York (1965)Google Scholar
- 2.C. K. Chow, An Optimum Character Recognition System Using Decision Functions, IRE Trans. on Electronic Computers,
**EC-6**, 247–254 (Dec. 1957)Google Scholar - 3.K. S. Fu, A Sequential Decision Model for Optimum Recognition, in “Biological Prototypes and Synthetic Systems,“ Vol. I, Plenum Press, New York (1962)Google Scholar
- 4.Y. T. Chien and K. S. Fu, A Modified Sequential Recognition Machine Using Time-Varying Stopping Boundaries, IEEE Trans. on Information Theory IT-12, 206–214 (April 1966)CrossRefGoogle Scholar
- 5.K. S. Fu, Y. T. Chien, and G. P. Cardillo, A Dynamic Programming Approach to Sequential Pattern Recognition, IEEE Trans. on Electronic Computers,
**Dec. 1967**, 790–803Google Scholar - 6.F. W. Smith, Contact Control by Adaptive Pattern-Recognition Techniques, Tech. Rept. No. 6762–2, Stanford Electronics Laboratories, Stanford, California (April 1964)Google Scholar
- 7.B. Widrow and F. W. Smith, Pattern Recognizing Control Systems, in: “Computer and Information Sciences,“ (J. T. Tou and R. H. Wilcox, eds.), Spartan Books, Washington, D. C. (1964)Google Scholar
- 8.R. Bush and F. Mosteller, “Stochastic Models for Learning,“ John Wiley and Sons, New York (1955)Google Scholar
- 9.M. L. Tsetlin, On the Behavior of Finite Automata in Random Environments, Avtomatika i Telemekhanika 22 (10), 1345–1354 (1961)Google Scholar
- 10.V. I. Varshavskii and I. P. Vorontsova, On the Behavior of Stochastic Automata with Variable Structure, Avtomatika i Telemekhanika
**24**(3), 353–360 (1963)Google Scholar - 11.K. S. Fu and R. W. McLaren, An Application of Stochastic Automata to the Synthesis of Learning Systems, Technical Report TR-EE-65-17, School of Electrical Engineering, Purdue University (Sept. 1965)Google Scholar
- 12.G. J. McMurtry and K. S. Fu, A Variable Structure Automaton Used as a Multi-Modal Searching Technique, IEEE Trans. on Automatic Control AC-11, 379–387 (July 1966)CrossRefGoogle Scholar
- 13.K. S. Fu and Z. J. Nikolic, On Some Reinforcement Techniques and their Relations With Stochastic Approximation, IEEE Trans. on Automatic Control AC-11 (Oct. 1966)Google Scholar
- 14.M. D. Waltz and K. S. Fu, A Heuristic Approach to Reinforcement Learning Control Systems, IEEE Trans. on Automatic Control
**AC-10**, 390–398 (Oct. 1965)CrossRefGoogle Scholar - 15.J. M. Mendel, Survey of Learning Control Systems for Space Vehicle Applications, Preprints, 1966 JACC (Aug. 1966)Google Scholar
- 16.H. Freeman, On the Digital Computer Classification of Geometric Line Patterns, in “Proc. National Electronics Conference,” Vol. 18 (Oct. 1962)Google Scholar
- 17.J. T. Tou, “Modern Control Theory,” McGraw-Hill Book Co., New York (1964)Google Scholar
- 18.J. C. Hsu and W. E. Meserve, Decision-Making in Adaptive Control Systems, IRE Trans. on Automatic Control Jan. 1962, 24–32Google Scholar
- 19.N. Ula and M. Kim, An Empirical Bayes Approach to Adaptive Control, J. Franklin Institute, 280 (3) (Sept. 1965)Google Scholar
- 20.Y. Sawaragi, Y. Sunahara, and T. Nakamizo, “Statistical Decision Theory in Adaptive Control Systems,” Academic Press, New York (1967)Google Scholar
- 21.K. S. Fu, A Class of Learning Control Systems Using Statistical Decision Functions, in “Proceedings of the Second IFAC (Teddington) Symposium on the Theory of Self-Adaptive Control Systems, Sept. 1965,“ Plenum Press, New York (1967)Google Scholar
- 22.N. Abramson and D. Braverman, Learning to Recognize Patterns in a Random Environment, IRE Trans. on Information Theory IT-8, 58–63 (Sept. 1962)CrossRefGoogle Scholar
- 23.D. G. Keehn, A Note on Learning for Gaussian Properties, IEEE Trans. on Information Theory
**IT-11**, 126–132 (Jan. 1965)CrossRefGoogle Scholar - 24.S. C. Fralick, Learning to Recognize a Pattern Without a Teacher, IEEE Trans. on Information Theory
**IT-13**, 57–64 (Jan. 1967)CrossRefGoogle Scholar - 25.H. Teicher, Identifiability of Finite Mixtures, Ann. Math. Stat. 34, 1265–1269 (Dec. 1963)CrossRefGoogle Scholar
- 26.Y. T. Chien and K. S. Fu, On Bayesian Learning and Stochastic Approximation, IEEE Trans. on System Science and Cybernetics
**June 1967**, 28–38Google Scholar - 27.Z. J. Nikolic and K. S. Fu, On the Estimation and Decomposition of Mixture Using Stochastic Approximation, in: “Proc. 1967 SWIEEECO, April 1967.”Google Scholar
- 28.Z. J. Nikolic and K. S. Fu, An Algorithm for Learning Without External Supervision and Its Application to Learning Control Systems, IEEE Trans. on Automatic Control,
**AC-11**, 414–422 (July 1966)CrossRefGoogle Scholar - 29.Ya. Z. Tsypkin, Adaptation, Learning and Self-Learning in Control Systems, paper presented at Third IFAC Congress, London (June 20–25, 1966)Google Scholar
- 30.K. S. Fu and Z. J. Nikolic, A Study of Learning Systems Operating in Unknown Stationary Environments, Technical Report TR-EE-66-20, School of Electrical Engineering, Purdue University (Nov. 1966)Google Scholar
- 31.M. A. Aizerman, E. M. Braverman, and L. I. Rozonoer, Theoretical Foundations of the Potential Function Method in Pattern Recognition, Avtomatika i Telemekhanika
**25**, 917–936 (1964)Google Scholar - 32.M. A. Aizerman, E. M. Braverman, and L. I. Rozonoer, The Probability Problem of Pattern Recognition Learning and the Method of Potential Functions, Avtomatika i Telemekhanika
**25**, 1307–1323 (1964)Google Scholar - 33.M. A. Aizerman. E. M. Braverman, and L. I. Rozonoer, The Method of Potential Functions for the Problem of Restoring the Characteristic of a Function Converter from Randomly Observed Points, Avtomatika i Telemekhanika
**25**, 1705–2213 (1964)Google Scholar - 34.E. M. Braverman and E. S. Pyatnitskii, Estimation of the Rate of Convergence of Algorithms Based on the Potential Function Method, Avtomatika i Telemakhanika,
**27**, 95–112 (1966)Google Scholar - 35.G. J. Simmons, Iterative Storage of Multidimensional Functions in Discrete Distributed Memories, in “Computer and Information Sciences - II” (J. T. Tou, ed.), pp. 261–280, Academic Press, New York (1967)Google Scholar
- 36.N. Aronszajn, Theory of Reproducing Kernels, Trans. Am. Math. Soc. 68, 337–404 (1950)CrossRefGoogle Scholar
- 37.F. G. Tricomi, “Integral Equations,” John Wiley and Sons, New York (1957)Google Scholar
- 38.Z. J. Nikolic and K. S. Fu, A Mathematical Model of Learning in an Unknown Random Environment, in: “Proc. National Electronics Conference,“ Vol. 22, 607–612, NEC (1966)Google Scholar
- 39.H. Robbins and S. Monro, A Stochastic Approximation Method, Ann. Math. Stat.
**22**(1), 400–407 (1951)CrossRefGoogle Scholar - 40.J. A. Blum, Approximation Methods which Converge with Probability One, Ann. Math. Stat.
**25**(2), 382–386 (1954)CrossRefGoogle Scholar - 41.J. A. Blum, Multidimensional Stochastic Approximation Procedures, Ann. Math. Stat.
**25**(4), 737–744 (1954)CrossRefGoogle Scholar - 42.E. G. Gladyshev, On Stochastic Approximation, Teoriya Veroyatnostei i ee Primeneniya
**10**(2) (1965)Google Scholar - 43.M. Driml and N. Nedoma, Stochastic Approximations for Continuous Random Processes, in “Trans. Second Prague Conference on Information Theory, Statistical Decision Functions, and Random Processes, 1960Google Scholar
- 44.O. Hans and A. Spacek, Random Fixed Point Approximation by Differentiable Trajectories, in “Trans. Second Prague Conference on Information Theory, Statistical Decision Functions, and Random Processes (1960)Google Scholar
- 45.M. Driml and O. Hans, Continuous Stochastic Approximations, in “Trans. Second Prague Conference on Information Theory, Statistical Decision Functions, and Random Processes (1960)Google Scholar
- 46.J. Kiefer and J. Wolfowitz, Stochastic Estimation of the Maximum of a Regression Function, Ann. Math. Stat.,
**23**(3), 462–466 (1952)CrossRefGoogle Scholar - 47.J. A. Blum, A Note on Stochastic Approximation, Proc. Am. Math. Soc. 9, 404–407 (1958)CrossRefGoogle Scholar
- 48.J. Sacks, Asymptotic Distribution of Stochastic Approximations, Ann. Math. Stat.
**29**(2), 373–405 (1958)CrossRefGoogle Scholar - 49.D. L. Sakrison, A Continuous Kiefer-Wolfowitz Procedure for Random Processes, Ann. Math. Stat.
**35**(2), 590–599 (1964)CrossRefGoogle Scholar - 50.A. Dvoretzky, On Stochastic Approximation, in “Proc. Third Berkeley Symposium on Mathematical Statistics and Probability, “Vol. 1 (1956)Google Scholar
- 51.K. B. Gray, Application of Stochastic Approximation to the Optimization of Random Circuits, in “Proc. Sixteenth Symposium on Applied Mathematics,“ Vol. XVI (1964)Google Scholar
- 52.H. Kesten, Accelerated Stochastic Approximation, Ann. Math. Stat.
**29**(1), 41–59 (1958)CrossRefGoogle Scholar - 53.V. Fabian, Stochastic Approximation Methods, Czech. Math. J.
**10**(1), 123–159 (1960)Google Scholar - 54.D. Burkholder, On A Class of Stochastic Approximation Processes, Ann. Math. Stat.
**27**(4), 1044–1059 (1956)CrossRefGoogle Scholar - 55.H. D. Block, Estimates of Error for Two Modifications of the Robbins-Monro Stochastic Approximation Process, Ann. Math. Stat.
**28**(4) (1957)CrossRefGoogle Scholar - 56.J. H. Venter, An Extension of the Robbins-Monro Procedure, Ann. Math. Stat.
**38**(2), 181–190 (1967)CrossRefGoogle Scholar - 57.V. Fabian, Stochastic Approximation of Minima with Improved Asymptotic Speed, Ann. Math. Stat.
**38**(2) 191–200 (1967)CrossRefGoogle Scholar - 58.K. S. Fu, Z. J. Nikolic, Y. T. Chien, and W. G. Wee, On the Stochastic Approximation and Related Learning Techniques, Technical Report TR-EE-66-6, School of Electrical Engineering, Purdue University (April 1966)Google Scholar
- 59.K. S. Fu, Learning Control Systems, in “Computer and Information Sciences (J. T. Tou and R. H. Wilcox, eds.), Spartan Books, Washington, D. C. (1964)Google Scholar
- 60.J. Sklansky, Learning Systems for Automatic Control, IEEE Trans. on Automatic Control AC-11 (Jan. 1966)Google Scholar
- 61.J. T. Tou and J. D. Hill, Steps Toward Learning Control, Preprints, 1966 JACC (Aug. 1966)Google Scholar
- 62.K. S. Fu, Stochastic Automata as Models of Learning Systems, in “Computer and Information Sciences-II,” (J. T. Tou, ed.), Academic Press, New York (1967)Google Scholar
- 63.R. W. McLaren, A Stochastic Automaton Model for Synthesis of Learning Systems, IEEE Trans. on System Science and Cybernetics
**SSC-2**, 109–114 (Dec. 1966)Google Scholar - 64.G. D. Bruce and K. S. Fu, A Model for Finite-State Probabilistic Systems, in “Proc. of the First Allerton Conference on Circuit and System Theory,“ University of Michigan Press (1963)Google Scholar
- 65.J. W. Carlyle, Equivalent Stochastic Sequential Machines, Technical Report, Series 60, Issue 415, Electronics Research Laboratory, University of California, Berkeley (1961)Google Scholar
- 66.J. W. Carlyle, Reduced Forms for Stochastic Sequential Machines, j. Math. Analysis and Application
**I**, 167–175 (1963)CrossRefGoogle Scholar - 67.G. C. Bacon, The Decomposition of Stochastic Automata, Information and Control
**I**, 320–339 (1964)CrossRefGoogle Scholar - 68.G. C. Bacon, Minimal-State Stochastic Finite-State Systems, IEEE Trans. on Circuit Theory
**CT-11**, 307–308 (1964)Google Scholar - 69.M. O. Rabin, Probabilistic Automata, Information and Control
**6**, 230–245 (1963)CrossRefGoogle Scholar - 70.A. Paz, Some Aspects of Probabilistic Automata, Information and Control
**9**(1), 26–60 (1966)CrossRefGoogle Scholar - 71.G. N. Tsertsvadze, Stochastic Automata and the Problem of Constructing Reliable Automata of Unreliable Elements, Avtomatika i Telemekhanika
**25**(1964)Google Scholar - 72.H. E. Maurer, An Approach to the Design of Reliable Radiation Hardened Integrated Logic and Sequential Circuits, Ph. D. Thesis, Purdue University (Aug. 1965)Google Scholar
- 73.G. N. Tsertsvadze, Certain Properties of Stochastic Automata and Methods, for Synthesizing Them, Avtomatika i Telemekhanika
**24**, 353–360 (1963)Google Scholar - 74.M. L. Tsetlin and V. Ju. Krylov, On the Automata Games, Avtomatika i Telemekhanika
**24**(1963)Google Scholar - 75.M. L. Tsetlin, S. L. Ginzburg, and V. Ju. Krylov, On an Example of a Many-Automata Games, Avtomatika i Telemekhanika
**26**(1965)Google Scholar - 76.R. L. Kashyap, Optimization of Stochastic Finite State Systems, paper presented of the Fifth Symposium on Discrete Adaptive Processes, Chicago, Illinois, (Oc. 3–5, 1966)Google Scholar
- 77.K. S. Fu and G. J. McMurtry, A Study of Stochastic Automata as Models of Adaptive and Learning Controllers, Rept. TR-EE65–17, Purdue University (June 1965)Google Scholar
- 78.B. Chandrasekaran and D. W. C. Shen, On Expediency and Convergence in Variable Structure Automata, paper presented at the Fifth Symposium on Discrete Adaptive Process, Chicago, Illinois (Oct. 3–5, 1966)Google Scholar
- 79.K. S. Fu, On Learning Techniques in Engineering Cybernetic Systems, in: “Proc. Vth International Congress on Cybernetics, Sept. 11–14, 1967, Namur, Belgium.”Google Scholar