# Control and Machine Intelligence for System Autonomy

- 244 Downloads

## Abstract

Autonomous systems evolve from control systems by adding functionalities that increase the level of system autonomy. It is very important to the research in the field that autonomy be well defined and so in the present paper a precise, useful definition of autonomy is introduced and discussed. Autonomy is defined as the ability of the system to attain a set of goals under a set of uncertainties. This leads to the notion of degrees or levels of autonomy. The Quest for Autonomy in engineered systems throughout the centuries is noted, connections to research work of 30 years ago are made and a hierarchical functional architecture for autonomous systems together with needed functionalities are outlined. Adaptation and Learning, which are among the most important functions in achieving high levels of autonomy are then highlighted and recent research contributions are briefly discussed.

## Keywords

Autonomy Machine intelligence Adaptation and learning## Notes

## References

- 1.Antsaklis, P.: Control systems and the quest for autonomy, Editorial. IEEE Trans. Autom. Control
**62**(3), 1013–1016 (2017)MathSciNetCrossRefzbMATHGoogle Scholar - 2.Antsaklis, P.J.: Defining intelligent control. IEEE Control Systems Society Report of the Task Force on Intelligent Control. IEEE Control. Syst.
**14**(3), 4–5, 58–66 (1994)Google Scholar - 3.Antsaklis, P.J.: On intelligent control: report of the IEEE CSS task force on intelligent control. Technical Report of the Interdisciplinary Studies of Intelligent Systems Group. University of Notre Dame
**94**, 001 (1994)Google Scholar - 4.Antsaklis, P.J.: Intelligent learning control. Introduction to Special Issue, IEEE Control. Syst.
**15**(3), 5–7 (1995)Google Scholar - 5.Antsaklis, P.J.: Intelligent control. Wiley Encyclopedia of Electrical and Electronics Engineering (1999)Google Scholar
- 6.Antsaklis, P.J.: The quest for autonomy revisited. Technical Report of the Interdisciplinary Studies of Intelligent Systems Group, University of Notre Dame
**11**, 004 (2011)Google Scholar - 7.Antsaklis, P.J., Passino, K.: Autonomous control systems: Architecture and concepts for future space vehicles. Final Report, Contract 957856, Jet Propulsion Laboratory (1987)Google Scholar
- 8.Antsaklis, P.J., Passino, K.M.: Introduction to intelligent control systems with high degrees of autonomy. Kluwer Academic Publishers (1993)Google Scholar
- 9.Antsaklis, P.J., Passino, K.M., Wang, S.: Towards intelligent autonomous control systems: architecture and fundamental issues. J. Intell. Robot. Syst.
**1**(4), 315–342 (1989)CrossRefGoogle Scholar - 10.Antsaklis, P.J., Passino, K.M., Wang, S.: An introduction to autonomous control systems. IEEE Control. Syst.
**11**(4), 5–13 (1991)CrossRefGoogle Scholar - 11.Åström, K.J., Wittenmark, B.: Adaptive control. Courier Corporation (2013)Google Scholar
- 12.Aström, K.J., Albertos, P., Blanke, M., Isidori, A., Schaufelberger, W., Sanz, R.: Control of complex systems. Springer, Berlin (2011)Google Scholar
- 13.Barto, A.G., Bradtke, S.J., Singh, S.P.: Learning to act using real-time dynamic programming. Artif. Intell.
**72**(1-2), 81–138 (1995)CrossRefGoogle Scholar - 14.Bcrtsekas, D.: Dynamic programming and optimal control, vol. I. Athena Scientific, Bellmont (1995)Google Scholar
- 15.Benard, N., Pons-Prat, J., Periaux, J., Bugeda, G., Bonnet, J.P., Moreau, E.: Multi-input genetic algorithm for experimental optimization of the reattachment downstream of a backward-facing-step with surface plasma actuator. In: 46th AIAA Plasmadynamics and lasers conference, pp. 2957–2980 (2015)Google Scholar
- 16.Bertsekas, D.P., Tsitsiklis, J.N.: Neuro-Dynamic programming: an overview. In: Proceedings of the 34Th IEEE conference on decision and control, IEEE, vol. 1, pp 560–564 (1995)Google Scholar
- 17.Bukkems, B., Kostic, D., De Jager, B., Steinbuch, M.: Learning-based identification and iterative learning control of direct-drive robots. IEEE Trans. Control Syst. Technol.
**13**(4), 537–549 (2005)CrossRefGoogle Scholar - 18.Chi, R., Liu, X., Zhang, R., Hou, Z., Huang, B.: Constrained data-driven optimal iterative learning control. J. Process. Control
**55**, 10–29 (2017)CrossRefGoogle Scholar - 19.Chowdhary, G.V., Johnson, E.N.: Theory and flight-test validation of a concurrent-learning adaptive controller. J. Guid. Control. Dyn.
**34**(2), 592–607 (2011)CrossRefGoogle Scholar - 20.Dai, S.L., Wang, C., Wang, M.: Dynamic learning from adaptive neural network control of a class of nonaffine nonlinear systems. IEEE Transactions on Neural Networks and Learning Systems
**25**(1), 111–123 (2014)CrossRefGoogle Scholar - 21.Doya, K.: Reinforcement learning in continuous time and space. Neural Comput.
**12**(1), 219–245 (2000)CrossRefGoogle Scholar - 22.Dracopoulos, D.C.: Genetic algorithms and genetic programming for control. In: Evolutionary algorithms in engineering applications, pp. 329–343. Springer (1997)Google Scholar
- 23.Feng, L., Zhang, K., Chai, Y., Yang, Z., Xu, S.: Observer-based fault estimators using iterative learning scheme for linear time-delay systems with intermittent faults. Asian J. Control
**19**(6), 1991–2008 (2017)MathSciNetCrossRefzbMATHGoogle Scholar - 24.Foroutan, S.A., Salmasi, F.R.: Detection of false data injection attacks against state estimation in smart grids based on a mixture gaussian distribution learning method. IET Cyber-Physical Systems: Theory & Applications
**2**(4), 161–171 (2017)Google Scholar - 25.Fu, K.S.: Learning control systems–review and outlook. IEEE Trans. Autom. Control
**15**(2), 210–221 (1970)MathSciNetCrossRefGoogle Scholar - 26.Goebel, G., Allgöwer, F: Semi-explicit mpc based on subspace clustering. Automatica
**83**, 309–316 (2017)MathSciNetCrossRefzbMATHGoogle Scholar - 27.Hein, D., Hentschel, A., Runkler, T., Udluft, S.: Particle swarm optimization for generating interpretable fuzzy reinforcement learning policies. Eng. Appl. Artif. Intel.
**65**, 87–98 (2017)CrossRefGoogle Scholar - 28.Hu, J., Zhou, M., Li, X., Xu, Z.: Online model regression for nonlinear time-varying manufacturing systems. Automatica
**78**, 163–173 (2017)MathSciNetCrossRefzbMATHGoogle Scholar - 29.Kamalapurkar, R., Reish, B., Chowdhary, G., Dixon, W.E.: Concurrent learning for parameter estimation using dynamic state-derivative estimators. IEEE Trans. Autom. Control
**62**(7), 3594–3601 (2017)MathSciNetCrossRefzbMATHGoogle Scholar - 30.Kiumarsi, B., Lewis, F.L., Jiang, Z.P.: H\(\infty \) control of linear discrete-time systems: Off-policy reinforcement learning. Automatica
**78**, 144–152 (2017)MathSciNetCrossRefzbMATHGoogle Scholar - 31.Kiumarsi, B., Vamvoudakis, K.G., Modares, H., Lewis, F.L.: Optimal and autonomous control using reinforcement learning: A survey. IEEE Transactions on Neural Networks and Learning Systems (2017)Google Scholar
- 32.Kokar, M.: Machine learning in a dynamic world. In: Proceedings of IEEE international symposium on intelligent control, pp. 500–507. IEEE (1988)Google Scholar
- 33.Lagoudakis, M.G., Parr, R., Littman, M.L.: Least-squares methods in reinforcement learning for control. In: Hellenic conference on artificial intelligence, pp. 249–260. Springer (2002)Google Scholar
- 34.Lee, C., Kim, J., Babcock, D., Goodman, R.: Application of neural networks to turbulence control for drag reduction. Phys. Fluids
**9**(6), 1740–1747 (1997)CrossRefGoogle Scholar - 35.Lewis, F.L., Vrabie, D., Syrmos, V.L.: Optimal control. Wiley , Hoboken (2012)CrossRefzbMATHGoogle Scholar
- 36.Lewis, F.L., Vrabie, D., Vamvoudakis, K.G.: Reinforcement learning and feedback control: using natural decision methods to design optimal adaptive controllers. IEEE Control. Syst.
**32**(6), 76–105 (2012)MathSciNetCrossRefGoogle Scholar - 37.Michalewicz, Z., Janikow, C.Z., Krawczyk, J.B.: A modified genetic algorithm for optimal control problems. Computers & Mathematics with Applications
**23**(12), 83–94 (1992)CrossRefzbMATHGoogle Scholar - 38.Michalski, R.S., Carbonell, J.G., Mitchell, T.M.: Machine learning: an artificial intelligence approach. Springer, Berlin (2013)zbMATHGoogle Scholar
- 39.Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A.A., Veness, J., Bellemare, M.G., Graves, A., Riedmiller, M., Fidjeland, A.K., Ostrovski, G., et al.: Human-level control through deep reinforcement learning. Nature
**518**(7540), 529 (2015)CrossRefGoogle Scholar - 40.Mohri, M., Rostamizadeh, A., Talwalkar, A.: Foundations of machine learning. MIT Press, Cambridge (2012)zbMATHGoogle Scholar
- 41.Nageshrao, S.P., Lopes, G.A., Jeltsema, D., Babuška, R.: Port-hamiltonian systems in adaptive and learning control: a survey. IEEE Trans. Autom. Control
**61**(5), 1223–1238 (2016)MathSciNetCrossRefzbMATHGoogle Scholar - 42.Nedić, A, Olshevsky, A., Uribe, C.A.: Fast convergence rates for distributed non-bayesian learning. IEEE Trans. Autom. Control
**62**(11), 5538–5553 (2017)MathSciNetCrossRefzbMATHGoogle Scholar - 43.Plaat, A., Kosters, W., van den Herik, J.: Computers and games. Springer, Berlin (2017)zbMATHGoogle Scholar
- 44.Sklansky, J.: Learning systems for automatic control. IEEE Trans. Autom. Control
**11**(1), 6–19 (1966)MathSciNetCrossRefGoogle Scholar - 45.Sutton, R.S., Barto, A.G.: Reinforcement learning: an introduction. 1. MIT Press, Cambridge (1998)Google Scholar
- 46.Sutton, R.S., Barto, A.G., Williams, R.J.: Reinforcement learning is direct adaptive optimal control. IEEE Control. Syst.
**12**(2), 19–22 (1992)CrossRefGoogle Scholar - 47.Tsypkin, Y.: Self-learning–what is it? IEEE Trans. Autom. Control
**13**(6), 608–612 (1968)MathSciNetCrossRefGoogle Scholar - 48.Vrabie, D., Lewis, F.: Neural network approach to continuous-time direct adaptive optimal control for partially unknown nonlinear systems. Neural Netw.
**22**(3), 237–246 (2009)CrossRefzbMATHGoogle Scholar - 49.Vrabie, D., Vamvoudakis, K.G., Lewis, F.L.: Optimal adaptive control and differential games by reinforcement learning principles, vol. 2. IET (2013)Google Scholar
- 50.Werbos, P.J.: Neural networks for control and system identification. In: Proceedings of the 28th IEEE conference on decision and control, pp. 260–265. IEEE (1989)Google Scholar
- 51.Kraft, L. G., Campagna, D.: A summary comparison of CMAC neural network and traditional adaptive control systems. Neural Networks for Control, W. T. Miller, R. Sutton, and P. Werbos, MIT Press, Cambridge, MA (1990)Google Scholar
- 52.Xie, J., Wan, Y., Mills, K., Filliben, J.J., Lewis, F.: A scalable sampling method to high-dimensional uncertainties for optimal and reinforcement learning-based controls. IEEE Control Systems Letters
**1**(1), 98–103 (2017)CrossRefGoogle Scholar - 53.Yang, C., Teng, T., Xu, B., Li, Z., Na, J., Su, C.Y.: Global adaptive tracking control of robot manipulators using neural networks with finite-time learning convergence. Int. J. Control. Autom. Syst.
**15**(4), 1916–1924 (2017)CrossRefGoogle Scholar - 54.Yang, X., Ruan, X.: Reinforced gradient-type iterative learning control for discrete linear time-invariant systems with parameters uncertainties and external noises. IMA J. Math. Control. Inf.
**34**(4), 1117–1133 (2016)MathSciNetGoogle Scholar - 55.Yang, X., He, H., Liu, D., Zhu, Y.: Adaptive dynamic programming for robust neural control of unknown continuous-time non-linear systems. IET Control Theory Appl.
**11**(14), 2307–2316 (2017)MathSciNetCrossRefGoogle Scholar