Advertisement

Learning Control Systems

Chapter

Abstract

In designing an optimal control system, if all the a priori information about the controlled process (plant-environment) is known and can be described deterministically, the optimal controller is usually designed by deterministic optimization techniques. If all or a part of the a priori information can only be described statistically—for example, in terms of probability distribution or density functions—then stochastic or statistical design techniques will be used. However, if the a priori information required is unknown or incompletely known, in general an optimal design cannot be achieved. Two different approaches have been taken to solve this class of problems. One approach is to design a controller based only upon the amount of information available. In that case the unknown information is either ignored or is assumed to take on some known values chosen according to the designer’s best guess. The second approach is to design a controller which is capable of estimating the unknown information during its operation and of determining an optimal control action on the basis of the estimated information. In the first case a rather conservative design criterion (for example, the minimax criterion) is often used; the systems designed are in general inefficient and suboptimal. In the second case, if the estimated information gradually approaches the true information as time proceeds, then the controller thus designed will approach to the optimal controller.

Keywords

Stochastic Approximation Control Situation Switching Surface Reinforcement Algorithm Stochastic Automaton 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    N. J. Nilsson, “Learning Machines,“ McGraw-Hill Book Co., New York (1965)Google Scholar
  2. 2.
    C. K. Chow, An Optimum Character Recognition System Using Decision Functions, IRE Trans. on Electronic Computers, EC-6, 247–254 (Dec. 1957)Google Scholar
  3. 3.
    K. S. Fu, A Sequential Decision Model for Optimum Recognition, in “Biological Prototypes and Synthetic Systems,“ Vol. I, Plenum Press, New York (1962)Google Scholar
  4. 4.
    Y. T. Chien and K. S. Fu, A Modified Sequential Recognition Machine Using Time-Varying Stopping Boundaries, IEEE Trans. on Information Theory IT-12, 206–214 (April 1966)CrossRefGoogle Scholar
  5. 5.
    K. S. Fu, Y. T. Chien, and G. P. Cardillo, A Dynamic Programming Approach to Sequential Pattern Recognition, IEEE Trans. on Electronic Computers, Dec. 1967, 790–803Google Scholar
  6. 6.
    F. W. Smith, Contact Control by Adaptive Pattern-Recognition Techniques, Tech. Rept. No. 6762–2, Stanford Electronics Laboratories, Stanford, California (April 1964)Google Scholar
  7. 7.
    B. Widrow and F. W. Smith, Pattern Recognizing Control Systems, in: “Computer and Information Sciences,“ (J. T. Tou and R. H. Wilcox, eds.), Spartan Books, Washington, D. C. (1964)Google Scholar
  8. 8.
    R. Bush and F. Mosteller, “Stochastic Models for Learning,“ John Wiley and Sons, New York (1955)Google Scholar
  9. 9.
    M. L. Tsetlin, On the Behavior of Finite Automata in Random Environments, Avtomatika i Telemekhanika 22 (10), 1345–1354 (1961)Google Scholar
  10. 10.
    V. I. Varshavskii and I. P. Vorontsova, On the Behavior of Stochastic Automata with Variable Structure, Avtomatika i Telemekhanika 24 (3), 353–360 (1963)Google Scholar
  11. 11.
    K. S. Fu and R. W. McLaren, An Application of Stochastic Automata to the Synthesis of Learning Systems, Technical Report TR-EE-65-17, School of Electrical Engineering, Purdue University (Sept. 1965)Google Scholar
  12. 12.
    G. J. McMurtry and K. S. Fu, A Variable Structure Automaton Used as a Multi-Modal Searching Technique, IEEE Trans. on Automatic Control AC-11, 379–387 (July 1966)CrossRefGoogle Scholar
  13. 13.
    K. S. Fu and Z. J. Nikolic, On Some Reinforcement Techniques and their Relations With Stochastic Approximation, IEEE Trans. on Automatic Control AC-11 (Oct. 1966)Google Scholar
  14. 14.
    M. D. Waltz and K. S. Fu, A Heuristic Approach to Reinforcement Learning Control Systems, IEEE Trans. on Automatic Control AC-10, 390–398 (Oct. 1965)CrossRefGoogle Scholar
  15. 15.
    J. M. Mendel, Survey of Learning Control Systems for Space Vehicle Applications, Preprints, 1966 JACC (Aug. 1966)Google Scholar
  16. 16.
    H. Freeman, On the Digital Computer Classification of Geometric Line Patterns, in “Proc. National Electronics Conference,” Vol. 18 (Oct. 1962)Google Scholar
  17. 17.
    J. T. Tou, “Modern Control Theory,” McGraw-Hill Book Co., New York (1964)Google Scholar
  18. 18.
    J. C. Hsu and W. E. Meserve, Decision-Making in Adaptive Control Systems, IRE Trans. on Automatic Control Jan. 1962, 24–32Google Scholar
  19. 19.
    N. Ula and M. Kim, An Empirical Bayes Approach to Adaptive Control, J. Franklin Institute, 280 (3) (Sept. 1965)Google Scholar
  20. 20.
    Y. Sawaragi, Y. Sunahara, and T. Nakamizo, “Statistical Decision Theory in Adaptive Control Systems,” Academic Press, New York (1967)Google Scholar
  21. 21.
    K. S. Fu, A Class of Learning Control Systems Using Statistical Decision Functions, in “Proceedings of the Second IFAC (Teddington) Symposium on the Theory of Self-Adaptive Control Systems, Sept. 1965,“ Plenum Press, New York (1967)Google Scholar
  22. 22.
    N. Abramson and D. Braverman, Learning to Recognize Patterns in a Random Environment, IRE Trans. on Information Theory IT-8, 58–63 (Sept. 1962)CrossRefGoogle Scholar
  23. 23.
    D. G. Keehn, A Note on Learning for Gaussian Properties, IEEE Trans. on Information Theory IT-11, 126–132 (Jan. 1965)CrossRefGoogle Scholar
  24. 24.
    S. C. Fralick, Learning to Recognize a Pattern Without a Teacher, IEEE Trans. on Information Theory IT-13, 57–64 (Jan. 1967)CrossRefGoogle Scholar
  25. 25.
    H. Teicher, Identifiability of Finite Mixtures, Ann. Math. Stat. 34, 1265–1269 (Dec. 1963)CrossRefGoogle Scholar
  26. 26.
    Y. T. Chien and K. S. Fu, On Bayesian Learning and Stochastic Approximation, IEEE Trans. on System Science and Cybernetics June 1967, 28–38Google Scholar
  27. 27.
    Z. J. Nikolic and K. S. Fu, On the Estimation and Decomposition of Mixture Using Stochastic Approximation, in: “Proc. 1967 SWIEEECO, April 1967.”Google Scholar
  28. 28.
    Z. J. Nikolic and K. S. Fu, An Algorithm for Learning Without External Supervision and Its Application to Learning Control Systems, IEEE Trans. on Automatic Control, AC-11, 414–422 (July 1966)CrossRefGoogle Scholar
  29. 29.
    Ya. Z. Tsypkin, Adaptation, Learning and Self-Learning in Control Systems, paper presented at Third IFAC Congress, London (June 20–25, 1966)Google Scholar
  30. 30.
    K. S. Fu and Z. J. Nikolic, A Study of Learning Systems Operating in Unknown Stationary Environments, Technical Report TR-EE-66-20, School of Electrical Engineering, Purdue University (Nov. 1966)Google Scholar
  31. 31.
    M. A. Aizerman, E. M. Braverman, and L. I. Rozonoer, Theoretical Foundations of the Potential Function Method in Pattern Recognition, Avtomatika i Telemekhanika 25, 917–936 (1964)Google Scholar
  32. 32.
    M. A. Aizerman, E. M. Braverman, and L. I. Rozonoer, The Probability Problem of Pattern Recognition Learning and the Method of Potential Functions, Avtomatika i Telemekhanika 25, 1307–1323 (1964)Google Scholar
  33. 33.
    M. A. Aizerman. E. M. Braverman, and L. I. Rozonoer, The Method of Potential Functions for the Problem of Restoring the Characteristic of a Function Converter from Randomly Observed Points, Avtomatika i Telemekhanika 25, 1705–2213 (1964)Google Scholar
  34. 34.
    E. M. Braverman and E. S. Pyatnitskii, Estimation of the Rate of Convergence of Algorithms Based on the Potential Function Method, Avtomatika i Telemakhanika, 27, 95–112 (1966)Google Scholar
  35. 35.
    G. J. Simmons, Iterative Storage of Multidimensional Functions in Discrete Distributed Memories, in “Computer and Information Sciences - II” (J. T. Tou, ed.), pp. 261–280, Academic Press, New York (1967)Google Scholar
  36. 36.
    N. Aronszajn, Theory of Reproducing Kernels, Trans. Am. Math. Soc. 68, 337–404 (1950)CrossRefGoogle Scholar
  37. 37.
    F. G. Tricomi, “Integral Equations,” John Wiley and Sons, New York (1957)Google Scholar
  38. 38.
    Z. J. Nikolic and K. S. Fu, A Mathematical Model of Learning in an Unknown Random Environment, in: “Proc. National Electronics Conference,“ Vol. 22, 607–612, NEC (1966)Google Scholar
  39. 39.
    H. Robbins and S. Monro, A Stochastic Approximation Method, Ann. Math. Stat. 22 (1), 400–407 (1951)CrossRefGoogle Scholar
  40. 40.
    J. A. Blum, Approximation Methods which Converge with Probability One, Ann. Math. Stat. 25 (2), 382–386 (1954)CrossRefGoogle Scholar
  41. 41.
    J. A. Blum, Multidimensional Stochastic Approximation Procedures, Ann. Math. Stat. 25 (4), 737–744 (1954)CrossRefGoogle Scholar
  42. 42.
    E. G. Gladyshev, On Stochastic Approximation, Teoriya Veroyatnostei i ee Primeneniya 10 (2) (1965)Google Scholar
  43. 43.
    M. Driml and N. Nedoma, Stochastic Approximations for Continuous Random Processes, in “Trans. Second Prague Conference on Information Theory, Statistical Decision Functions, and Random Processes, 1960Google Scholar
  44. 44.
    O. Hans and A. Spacek, Random Fixed Point Approximation by Differentiable Trajectories, in “Trans. Second Prague Conference on Information Theory, Statistical Decision Functions, and Random Processes (1960)Google Scholar
  45. 45.
    M. Driml and O. Hans, Continuous Stochastic Approximations, in “Trans. Second Prague Conference on Information Theory, Statistical Decision Functions, and Random Processes (1960)Google Scholar
  46. 46.
    J. Kiefer and J. Wolfowitz, Stochastic Estimation of the Maximum of a Regression Function, Ann. Math. Stat., 23 (3), 462–466 (1952)CrossRefGoogle Scholar
  47. 47.
    J. A. Blum, A Note on Stochastic Approximation, Proc. Am. Math. Soc. 9, 404–407 (1958)CrossRefGoogle Scholar
  48. 48.
    J. Sacks, Asymptotic Distribution of Stochastic Approximations, Ann. Math. Stat. 29 (2), 373–405 (1958)CrossRefGoogle Scholar
  49. 49.
    D. L. Sakrison, A Continuous Kiefer-Wolfowitz Procedure for Random Processes, Ann. Math. Stat. 35 (2), 590–599 (1964)CrossRefGoogle Scholar
  50. 50.
    A. Dvoretzky, On Stochastic Approximation, in “Proc. Third Berkeley Symposium on Mathematical Statistics and Probability, “Vol. 1 (1956)Google Scholar
  51. 51.
    K. B. Gray, Application of Stochastic Approximation to the Optimization of Random Circuits, in “Proc. Sixteenth Symposium on Applied Mathematics,“ Vol. XVI (1964)Google Scholar
  52. 52.
    H. Kesten, Accelerated Stochastic Approximation, Ann. Math. Stat. 29 (1), 41–59 (1958)CrossRefGoogle Scholar
  53. 53.
    V. Fabian, Stochastic Approximation Methods, Czech. Math. J. 10 (1), 123–159 (1960)Google Scholar
  54. 54.
    D. Burkholder, On A Class of Stochastic Approximation Processes, Ann. Math. Stat. 27 (4), 1044–1059 (1956)CrossRefGoogle Scholar
  55. 55.
    H. D. Block, Estimates of Error for Two Modifications of the Robbins-Monro Stochastic Approximation Process, Ann. Math. Stat. 28 (4) (1957)CrossRefGoogle Scholar
  56. 56.
    J. H. Venter, An Extension of the Robbins-Monro Procedure, Ann. Math. Stat. 38 (2), 181–190 (1967)CrossRefGoogle Scholar
  57. 57.
    V. Fabian, Stochastic Approximation of Minima with Improved Asymptotic Speed, Ann. Math. Stat. 38 (2) 191–200 (1967)CrossRefGoogle Scholar
  58. 58.
    K. S. Fu, Z. J. Nikolic, Y. T. Chien, and W. G. Wee, On the Stochastic Approximation and Related Learning Techniques, Technical Report TR-EE-66-6, School of Electrical Engineering, Purdue University (April 1966)Google Scholar
  59. 59.
    K. S. Fu, Learning Control Systems, in “Computer and Information Sciences (J. T. Tou and R. H. Wilcox, eds.), Spartan Books, Washington, D. C. (1964)Google Scholar
  60. 60.
    J. Sklansky, Learning Systems for Automatic Control, IEEE Trans. on Automatic Control AC-11 (Jan. 1966)Google Scholar
  61. 61.
    J. T. Tou and J. D. Hill, Steps Toward Learning Control, Preprints, 1966 JACC (Aug. 1966)Google Scholar
  62. 62.
    K. S. Fu, Stochastic Automata as Models of Learning Systems, in “Computer and Information Sciences-II,” (J. T. Tou, ed.), Academic Press, New York (1967)Google Scholar
  63. 63.
    R. W. McLaren, A Stochastic Automaton Model for Synthesis of Learning Systems, IEEE Trans. on System Science and Cybernetics SSC-2, 109–114 (Dec. 1966)Google Scholar
  64. 64.
    G. D. Bruce and K. S. Fu, A Model for Finite-State Probabilistic Systems, in “Proc. of the First Allerton Conference on Circuit and System Theory,“ University of Michigan Press (1963)Google Scholar
  65. 65.
    J. W. Carlyle, Equivalent Stochastic Sequential Machines, Technical Report, Series 60, Issue 415, Electronics Research Laboratory, University of California, Berkeley (1961)Google Scholar
  66. 66.
    J. W. Carlyle, Reduced Forms for Stochastic Sequential Machines, j. Math. Analysis and Application I, 167–175 (1963)CrossRefGoogle Scholar
  67. 67.
    G. C. Bacon, The Decomposition of Stochastic Automata, Information and Control I, 320–339 (1964)CrossRefGoogle Scholar
  68. 68.
    G. C. Bacon, Minimal-State Stochastic Finite-State Systems, IEEE Trans. on Circuit Theory CT-11, 307–308 (1964)Google Scholar
  69. 69.
    M. O. Rabin, Probabilistic Automata, Information and Control 6, 230–245 (1963)CrossRefGoogle Scholar
  70. 70.
    A. Paz, Some Aspects of Probabilistic Automata, Information and Control 9 (1), 26–60 (1966)CrossRefGoogle Scholar
  71. 71.
    G. N. Tsertsvadze, Stochastic Automata and the Problem of Constructing Reliable Automata of Unreliable Elements, Avtomatika i Telemekhanika 25 (1964)Google Scholar
  72. 72.
    H. E. Maurer, An Approach to the Design of Reliable Radiation Hardened Integrated Logic and Sequential Circuits, Ph. D. Thesis, Purdue University (Aug. 1965)Google Scholar
  73. 73.
    G. N. Tsertsvadze, Certain Properties of Stochastic Automata and Methods, for Synthesizing Them, Avtomatika i Telemekhanika 24, 353–360 (1963)Google Scholar
  74. 74.
    M. L. Tsetlin and V. Ju. Krylov, On the Automata Games, Avtomatika i Telemekhanika 24 (1963)Google Scholar
  75. 75.
    M. L. Tsetlin, S. L. Ginzburg, and V. Ju. Krylov, On an Example of a Many-Automata Games, Avtomatika i Telemekhanika 26 (1965)Google Scholar
  76. 76.
    R. L. Kashyap, Optimization of Stochastic Finite State Systems, paper presented of the Fifth Symposium on Discrete Adaptive Processes, Chicago, Illinois, (Oc. 3–5, 1966)Google Scholar
  77. 77.
    K. S. Fu and G. J. McMurtry, A Study of Stochastic Automata as Models of Adaptive and Learning Controllers, Rept. TR-EE65–17, Purdue University (June 1965)Google Scholar
  78. 78.
    B. Chandrasekaran and D. W. C. Shen, On Expediency and Convergence in Variable Structure Automata, paper presented at the Fifth Symposium on Discrete Adaptive Process, Chicago, Illinois (Oct. 3–5, 1966)Google Scholar
  79. 79.
    K. S. Fu, On Learning Techniques in Engineering Cybernetic Systems, in: “Proc. Vth International Congress on Cybernetics, Sept. 11–14, 1967, Namur, Belgium.”Google Scholar

Copyright information

© Plenum Press 1969

Authors and Affiliations

  1. 1.Purdue UniversityLafayetteUSA

Personalised recommendations