Learning, control and hybrid systems pp 254-268 | Cite as

# Universal output prediction and nonparametric regression for arbitrary data

## Abstract

We construct a class of elementary nonparametric output predictors of an unknown discrete-time nonlinear fading memory system. Our algorithms predict asymptotically well for every bounded input sequence, every disturbance sequence in certain classes, and every linear or nonlinear system that is continuous and asymptotically time-invariant, causal, and with fading memory. The predictor is based on *k* _{ n }-nearest neighbor estimators from nonparametric statistics. It uses only previous input and noisy output data of the system without any knowledge of the structure of the unknown system, the bounds on the input, or the properties of noise. Under additional smoothness conditions we provide rates of convergence for the time-average errors of our scheme. Finally, we apply our results to the special case of stable LTI systems.

## Keywords

Prediction Error Input Sequence Near Neighbor Nonparametric Regression Input String## Preview

Unable to display preview. Download preview PDF.

## References

- [1]T.M. Cover, “Estimation by the nearest neighbor rule,”
*IEEE Trans. Information Theory*, vol. IT-14, pp. 50–55, Jan. 1968.CrossRefGoogle Scholar - [2]M.A. Dahleh, E.D. Sontag, D.N.C. Tse, and J.N. Tsitsiklis, “Worst-case identification of nonlinear fading memory systems,” Automatica, vol. 31, pp. 503–508, 1995.zbMATHCrossRefMathSciNetGoogle Scholar
- [3]M.A. Dahleh, T. Theodosopoulos, and J.N. Tsitsiklis, “The sample complexity of worst-case identification for f.i.r. linear systems,”
*Syst. Contr. Lett.*, vol. 20, pp. 157–166, March 1993.zbMATHCrossRefMathSciNetGoogle Scholar - [4]L. Devroye, “Necessary and sufficient conditions for the pointwise convergence of nearest neighbor regression function estimates,”
*Z. Wahrscheinlichkeitstheorie verw. Gebiete*vol. 61, pp. 467–481, 1982.zbMATHCrossRefMathSciNetGoogle Scholar - [5]R.M. Dudley,
*Real Analysis and Probability*, Chapman & Hall, 1989.Google Scholar - [6]J.D. Farmer and J.J. Sidorowich, “Exploiting chaos to predict the future and reduce noise,”
*Evolution, Learning, and Cognition*, pp. 265–289, World Scientific, Singapore, 1988.Google Scholar - [7]M. Feder, N. Merhav, and M. Gutman, “Universal prediction of individual sequences,”
*IEEE Trans. Information Theory*, vol. 38, pp. 1258–1270, July 1992.zbMATHCrossRefMathSciNetGoogle Scholar - [8]W. Greblicki, “Nonparametric identification of Wiener systems by orthogonal series,”
*IEEE Trans. Automatic Control*, vol. 30, pp. 2077–2086, 1994.CrossRefMathSciNetGoogle Scholar - [9]W. Greblicki and M. Pawlak, “Dynamic system identification with order statistics,”
*IEEE Trans. Information Theory*, vol. 40, pp. 1474–1489, 1994.zbMATHCrossRefGoogle Scholar - [10]L. Györfi, “The rate of convergence of
*k*_{n}-NN regression estimates and classification rules,”*IEEE Trans. Information Theory*, vol. IT-27, pp. 362–364, May 1981.CrossRefGoogle Scholar - [11]A.J. Helmicki, C.A. Jacobson, C.N. Nett, “Control Oriented System Identification: A Worst-Case/Deterministic Approach in H
_{∞},”*IEEE Trans. Automatic Control*, vol. 36, Oct. 1991.Google Scholar - [12]A. Krzyżak, “On estimation of a class of nonlinear systems by the kernel regression estimate,”
*IEEE Trans. Information Theory*, vol. 36, pp. 141–152, 1990.CrossRefGoogle Scholar - [13]A. Krzyżak, “Identification of nonlinear systems by recursive kernel regression estimates,”
*Int. J. Systems Sci.*, vol. 24, pp. 577–598, 1993.CrossRefGoogle Scholar - [14]S.R. Kulkarni, “Data-dependent nearest neighbor and kernel estimators consistent for arbitrary processes,” preprint.Google Scholar
- [15]S.R. Kulkarni and S.E. Posner, “Rates of convergence of nearest neighbor estimation under arbitrary sampling,”
*IEEE Trans. Information Theory*, pp. 1028–1039, July 1995.Google Scholar - [16]S.R. Kulkarni and S.E. Posner, “Nonparametric output prediction for nonlinear fading memory systems,” to appear
*IEEE Trans. Automatic Control*, Nov., 1998.Google Scholar - [17]L. Ljung,
*System Identification: Theory for the User*, Prentice-Hall, 1987.Google Scholar - [18]P.M. Mäkilä, “Robust identification and Galois sequences,”
*Int. J. Contr.*, vol. 54, pp. 1189–1200, 1991.zbMATHCrossRefGoogle Scholar - [19]G. Morvai, S. Yakowitz, and L. Györfi, “Nonparametric inferences for ergodic stationary time series,” preprint.Google Scholar
- [20]K. Poolla and A. Tikku, “On the time complexity of worst-case system identification,”
*IEEE Trans. Automatic Control*, vol. 39, pp. 944–950, May 1994.zbMATHCrossRefMathSciNetGoogle Scholar - [21]S.E. Posner, “Nonparametric estimation, regression, and prediction under minimal regularity conditions,” Ph.D. Thesis, Department of Electrical Engineering, Princeton University, 1995.Google Scholar
- [22]C.J. Stone, “Consistent nonparametric regression,”
*Ann. Stat.*, vol. 8, pp. 1348–1360, 1977.Google Scholar - [23]D.N.C. Tse, M.A. Dahleh, J.N. Tsitsiklis, “Optimal asymptotic identification under bounded disturbances,”
*IEEE Trans. Automatic Control*, vol. 38, pp. 1176–1190, Aug. 1993.zbMATHCrossRefMathSciNetGoogle Scholar