Universal output prediction and nonparametric regression for arbitrary data

  • S. R. Kulkarni
  • S. E. Posner
Part C Modeling, Identification And Estimation
Part of the Lecture Notes in Control and Information Sciences book series (LNCIS, volume 241)


We construct a class of elementary nonparametric output predictors of an unknown discrete-time nonlinear fading memory system. Our algorithms predict asymptotically well for every bounded input sequence, every disturbance sequence in certain classes, and every linear or nonlinear system that is continuous and asymptotically time-invariant, causal, and with fading memory. The predictor is based on k n -nearest neighbor estimators from nonparametric statistics. It uses only previous input and noisy output data of the system without any knowledge of the structure of the unknown system, the bounds on the input, or the properties of noise. Under additional smoothness conditions we provide rates of convergence for the time-average errors of our scheme. Finally, we apply our results to the special case of stable LTI systems.


Prediction Error Input Sequence Near Neighbor Nonparametric Regression Input String 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [1]
    T.M. Cover, “Estimation by the nearest neighbor rule,” IEEE Trans. Information Theory, vol. IT-14, pp. 50–55, Jan. 1968.CrossRefGoogle Scholar
  2. [2]
    M.A. Dahleh, E.D. Sontag, D.N.C. Tse, and J.N. Tsitsiklis, “Worst-case identification of nonlinear fading memory systems,” Automatica, vol. 31, pp. 503–508, 1995.zbMATHCrossRefMathSciNetGoogle Scholar
  3. [3]
    M.A. Dahleh, T. Theodosopoulos, and J.N. Tsitsiklis, “The sample complexity of worst-case identification for f.i.r. linear systems,” Syst. Contr. Lett., vol. 20, pp. 157–166, March 1993.zbMATHCrossRefMathSciNetGoogle Scholar
  4. [4]
    L. Devroye, “Necessary and sufficient conditions for the pointwise convergence of nearest neighbor regression function estimates,” Z. Wahrscheinlichkeitstheorie verw. Gebiete vol. 61, pp. 467–481, 1982.zbMATHCrossRefMathSciNetGoogle Scholar
  5. [5]
    R.M. Dudley, Real Analysis and Probability, Chapman & Hall, 1989.Google Scholar
  6. [6]
    J.D. Farmer and J.J. Sidorowich, “Exploiting chaos to predict the future and reduce noise,” Evolution, Learning, and Cognition, pp. 265–289, World Scientific, Singapore, 1988.Google Scholar
  7. [7]
    M. Feder, N. Merhav, and M. Gutman, “Universal prediction of individual sequences,” IEEE Trans. Information Theory, vol. 38, pp. 1258–1270, July 1992.zbMATHCrossRefMathSciNetGoogle Scholar
  8. [8]
    W. Greblicki, “Nonparametric identification of Wiener systems by orthogonal series,” IEEE Trans. Automatic Control, vol. 30, pp. 2077–2086, 1994.CrossRefMathSciNetGoogle Scholar
  9. [9]
    W. Greblicki and M. Pawlak, “Dynamic system identification with order statistics,” IEEE Trans. Information Theory, vol. 40, pp. 1474–1489, 1994.zbMATHCrossRefGoogle Scholar
  10. [10]
    L. Györfi, “The rate of convergence of k n-NN regression estimates and classification rules,” IEEE Trans. Information Theory, vol. IT-27, pp. 362–364, May 1981.CrossRefGoogle Scholar
  11. [11]
    A.J. Helmicki, C.A. Jacobson, C.N. Nett, “Control Oriented System Identification: A Worst-Case/Deterministic Approach in H ,” IEEE Trans. Automatic Control, vol. 36, Oct. 1991.Google Scholar
  12. [12]
    A. Krzyżak, “On estimation of a class of nonlinear systems by the kernel regression estimate,” IEEE Trans. Information Theory, vol. 36, pp. 141–152, 1990.CrossRefGoogle Scholar
  13. [13]
    A. Krzyżak, “Identification of nonlinear systems by recursive kernel regression estimates,” Int. J. Systems Sci., vol. 24, pp. 577–598, 1993.CrossRefGoogle Scholar
  14. [14]
    S.R. Kulkarni, “Data-dependent nearest neighbor and kernel estimators consistent for arbitrary processes,” preprint.Google Scholar
  15. [15]
    S.R. Kulkarni and S.E. Posner, “Rates of convergence of nearest neighbor estimation under arbitrary sampling,” IEEE Trans. Information Theory, pp. 1028–1039, July 1995.Google Scholar
  16. [16]
    S.R. Kulkarni and S.E. Posner, “Nonparametric output prediction for nonlinear fading memory systems,” to appear IEEE Trans. Automatic Control, Nov., 1998.Google Scholar
  17. [17]
    L. Ljung, System Identification: Theory for the User, Prentice-Hall, 1987.Google Scholar
  18. [18]
    P.M. Mäkilä, “Robust identification and Galois sequences,” Int. J. Contr., vol. 54, pp. 1189–1200, 1991.zbMATHCrossRefGoogle Scholar
  19. [19]
    G. Morvai, S. Yakowitz, and L. Györfi, “Nonparametric inferences for ergodic stationary time series,” preprint.Google Scholar
  20. [20]
    K. Poolla and A. Tikku, “On the time complexity of worst-case system identification,” IEEE Trans. Automatic Control, vol. 39, pp. 944–950, May 1994.zbMATHCrossRefMathSciNetGoogle Scholar
  21. [21]
    S.E. Posner, “Nonparametric estimation, regression, and prediction under minimal regularity conditions,” Ph.D. Thesis, Department of Electrical Engineering, Princeton University, 1995.Google Scholar
  22. [22]
    C.J. Stone, “Consistent nonparametric regression,” Ann. Stat., vol. 8, pp. 1348–1360, 1977.Google Scholar
  23. [23]
    D.N.C. Tse, M.A. Dahleh, J.N. Tsitsiklis, “Optimal asymptotic identification under bounded disturbances,” IEEE Trans. Automatic Control, vol. 38, pp. 1176–1190, Aug. 1993.zbMATHCrossRefMathSciNetGoogle Scholar

Copyright information

© Springer-Verlag London Limited 1999

Authors and Affiliations

  • S. R. Kulkarni
    • 1
  • S. E. Posner
    • 2
  1. 1.Department of Electrical EngineeringPrinceton UniversityPrinceton
  2. 2.ING Baring Securities, Inc.New York

Personalised recommendations