Abstract
In the simplest formulation, modelling from a time series is considered as restoration of an explicit temporal dependence \(\eta =f(t,\textbf{c})\), where f is a certain function and c is the P-dimensional vector of model parameters. Such a problem setting is considered in the theory of function approximation (Akhieser, 1965) and mathematical statistics (Aivazian, 1968; Hardle, 1992; Seber, 1977). It can be interpreted as drawing a curve through experimental data points on the plane \(\left({t,\eta}\right)\) or near those points (Fig. 7.1). A capability of solving this problem determines to a significant extent the success of modelling in more complex situations discussed in Chaps. 8, 9 and 10.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
An example. In the Ptolemaic astronomy, motions of celestial bodies relative to the Earth are approximated with combinations of motions along circumferences (epicycles), whose centres move along other circumferences around the Earth. To provide high accuracy in predictions of the future locations of planets on the coelosphere, which was important for sea navigation, a very sophisticated system of epicycles was developed. That work stimulated the development of the spherical geometry in the Middle Ages. When the geocentric system got an alternative, the heliocentric system of Copernicus, the latter one was inferior to its predecessor with respect to the prediction accuracy. With the further development of ideas and models of the heliocentric system, motions of planets got better described with it and it replaced the geocentric theory.
- 2.
Mathematical encyclopaedic dictionary gives a wider definition of interpolation as an approximate or accurate finding of some quantity from its known separate values (Mathematical dictionary, 1988). Under such a definition, interpolation covers even the notion of approximation. In the case of extrapolation, classical mathematics uses, vice versa, somewhat narrower meaning as a continuation of a function outside its domain such that the continued function (as a rule, an analytic one) belongs to a given class. We employ the definitions given in the main text, which are widely used by specialists in numerical analysis.
- 3.
The term was first used by English statistician F. Galton (1866). He studied how height of children Y depends on the height of their parents X and found the following. If the height of parents exceeds an average height of people by b, then the height of their children exceeds the average height by less than b. This phenomenon was called regression, i.e. “backward motion”. Therefore, dependence of the conditional mean of Y on X was also called regression.
- 4.
Even if the true regression \(F(t)\) belongs to the selected model class \(f(t,\textbf{c})\), i.e. \(F(t)=f(t,\textbf{c}_{0})\), restoration of \(F(t)\) is not equivalent to the most accurate estimation of the parameters c 0. The point is that the best estimate of the parameters in the “true” class \(f(t,\textbf{c})\) does not necessarily give the best approximation to F from a finite data sample, since the best approximation may be achieved in another class (Vapnik, 1979).
- 5.
Pluralitas non est ponenda sine necessitate. William Ockham (1285–1349) is a famous English philosopher and logician.
- 6.
Each value of \(\hat{\varepsilon}_{k}\) corresponds to some value of the empirical distribution function for the residuals \(\hat{\Phi}(\hat{\varepsilon}_{k})\). The reconstruction of a distribution function is a well-posed problem (Vapnik, 1979). Thus, if all \(\hat{\varepsilon}_{i}\) are pairwise different, then \(\hat{\Phi}(\hat{\varepsilon}_{k})\) can be estimated as the ratio of the number of values \(\hat{\varepsilon}_{i}\), less than or equal to \(\hat{\varepsilon}_{k}\), to their total number N. Let us denote the distribution function of the standard Gaussian law (zero mean and unit variance) as \(\Phi_{0}(x)\). Let us denote x k such a number that \(\Phi_{0}(x_{k})=\hat{\Phi}(\hat{\varepsilon}_{k})\). x k is unique since \(\Phi_{0}(x)\) is a continuous and strictly monotonous function. x k is related to \(\hat{\varepsilon}_{k}\) as \(x_{k}=\Phi_{0}^{-1}(\hat{\Phi}(\hat{\varepsilon}_{k}))\). The plot x k versus \(\hat{\varepsilon}_{k}\) is called a plot on the normal probability paper.
- 7.
Instead of the variance \(\sigma_{\xi}^{2}\) one may substitute its estimate \(\hat{\sigma}_{\xi}^{2}\) into equation (7.35). The variance of the parameter estimator \(\sigma_{\hat{c}}^{2}\) is usually proportional to \(\sigma_{\xi}^{2}\) and inversely proportional to time series length N or higher degrees of N (Sect. 7.1.2). Formulas for a more general case of several estimated parameters c are as follows. For a pseudo-linear model (7.31), the covariance matrix for the parameter estimators is given by \({\mathrm{Cov}}(\hat{\textbf{c}})=\hat{\sigma}_{\xi}^{2}(\textbf{A}^{\mathrm{T}}\textbf{A})^{-1}\), where \(A_{j,k}=\sum\limits_{i=1}^{N}\phi_{j}(t_{i})\phi_{k}(t_{i})\) (Gnedenko, 1950). Diagonal elements of \(\mathit{Cov}(\hat{\textbf{c}})\) are the variances of the parameter estimators \(\hat{\sigma}_{\hat{c}_{i}}^{2}=[\mathit{Cov}(\hat{\textbf{c}})]_{\mathit{ii}}\). For a model non-linear with respect to the parameters, the covariance matrix estimator is obtained as an inverse Hessian of the likelihood function: \(\mathit{Cov}(\hat{\textbf{c}})=\textbf{H}^{-1}(\hat{\textbf{c}})\), where \(H_{\mathit{ij}}(\hat{\textbf{c}})=-\partial^{2}\ln L(\hat{\textbf{c}})/\partial c_{i}\partial c_{j}\).
- 8.
It is for the same reason that the parameter estimator variance for a process (7.19) decreases faster (as \(1/N^{3}\)) with N (Sect. 7.1.2). Since the function f is sensitive to variations in the parameter value, the cost function S in equation (7.12) is also sensitive to them, since the squared value of \(\partial f(t,c)/\partial c|_{c=c_{0}}\) is the partial Fisher information determining the contribution of each new observation to the likelihood function.
References
Aguirre, L.A., Freitas, U.S., Letellier, C., Maquet, J.: Structure-selection techniques applied to continuous-time nonlinear models. Physica D. 158, 1–18 (2001)
Aivazian, S.A.: Statistical Investigation of Dependencies. Metallurgiya, Moscow, (in Russian) (1968)
Akaike, H.: A new look at the statistical identification model. IEEE Trans. Automatic Control. 19, 716–723 (1974)
Akhieser, N.I.: Lectures on Theory of Approximation, 2nd edn. Nauka, Moscow (1965). 1st edn. is translated into English: Ungar Publishing Co., New York (1956)
Bezruchko, B.P., Dikanev, T.V., Smirnov, D.A.: Role of transient processes for reconstruction of model equations from time series. Phys. Rev. E. 64, 036210 (2001)
Box, G.E.P., Jenkins, G.M.: Time Series Analysis. Forecasting and Control. Holden-Day, San Francisco (1970)
Casdagli, M.: Nonlinear prediction of chaotic time series. Physica D. 35, 335–356 (1989)
Crutchfield, J.P., McNamara, B.S.: Equations of motion from a data series. Complex Syst. 1, 417–452 (1987)
Danilov Yu.A., Safonov, V.L.: Usage of uniform function approximation in trigonometric basis to compute integrals and sums and to process experimental data. Preprint Institute Atomic Energy No. 4381/1. TsNIIAtomInform, Moscow, (in Russian) (1987)
Dennis, J., Schnabel, R.: Numerical Methods for Unconstrained Optimization and Nonlinear Equations. Prentice-Hall, Engle Wood Cliffs, NJ (1983)
Farmer, J.D., Sidorowich, J.J.: Predicting chaotic time series. Phys. Rev. Lett. 59, 845–848 (1987)
Golub, G.H., Van Loan, C.F.: Matrix Computations. 2nd edn. Johns Hopkins University Press, Baltimore (1989)
Green, P.J., Silverman, B.W.: Nonparametric Regression and Generalized Linear Models. Chapman and Hall, London (1994)
Hardle, W.: Applied Nonparametric Regression. Cambridge University. Press, Cambridge (1992)
Hastie, T.J., Tibshirani, R.J.: Generalized Additive Models. Chapman and Hall, London (1990)
Hoel, P.G.: Introduction to Mathematical Statistics. 4th edn. Wiley, New York (1971)
Ibragimov, I.A., Has’minskii R.Z.: Asymptotic Theory of Estimation. Nauka, Moscow (1979). Translated into English Under the Title Statistical Estimation: Springer, New York (1981)
Johnson, L.W., Riess, R.D.: Numerical Analysis. 2nd edn. Addison-Wesley, Reading, MA (1982)
Judd, K., Mees, A.I. On selecting models for nonlinear time series. Physica D. 82, 426–444 (1995)
Judd, K., Mees, A.I.: Embedding as a modeling problem. Phys. D. 120, 273–286 (1998)
Judd, K., Small, M.: Towards long-term prediction. Phys. D. 136, 31–44 (2000)
Kalitkin, N.N.: Numerical Methods. Nauka, Moscow, (in Russian) (1978)
Kendall, M.G., Stuart, A.: The Advanced Theory of Statistics, vol. 2 and 3. Charles Griffin, London (1979)
Makarenko, N.G.: Embedology and neuro-prediction. Procs. V All-Russian Conf. “Neuroinformatics-2003”. Part 1, pp. 86–148. Moscow, (in Russian) (2003)
Nakamura, T., Kilminster, D., Judd, K., Mees, A. A comparative study of model selection methods for nonlinear time series. Int. J. Bif. Chaos. 14, 1129–1146 (2004)
Pisarenko, V.F., Sornette, D.: Statistical methods of parameter estimation for deterministically chaotic time series. Phys. Rev. E. 69, 036122 (2004)
Press, W.H., Flannery, B.P., Teukolsky, S.A., Vetterling, W.T.: Numerical Recipes in C. Cambridge University Press, Cambridge (1988)
Pugachev, V.S.: Probability Theory and Mathematical Statistics for Engineers. Pergamon Press, Oxford (1984)
Pugachev, V.S.: Theory of Probabilities and Mathematical Statistics. Nauka, Moscow, (in Russian) (1979)
Reinsch, C.H.: Smoothing by spline functions. Num. Math. 10, 177–183 (1967)
Rissanen, J. A universal prior for integers and estimation by minimum description length. Ann. Stat. 11, 416–431 (1993)
Rissanen, J.: Stochastic complexity in statistical inquiry. World Scientific, Singapore (1989)
Samarsky, A.A.: Introduction to Numerical Methods. Nauka, Moscow, (in Russian) (1982)
Schwarz, G.: Estimating the order of a model. Ann. Stat. 6, 461–464 (1978)
Seber, G.A.F.: Linear Regression Analysis. Wiley, New York (1977)
Shalizi, C.R.: Methods and techniques of complex systems science: an overview, vol. 3, arXiv:nlin.AO/0307015 (2003). Available at http://www.arxiv.org/abs/nlin.AO/0307015
Small, M., Judd, K.: Comparisons of new nonlinear modeling techniques with applications to infant respiration. Phys. D. 117, 283–298 (1998)
Stoer, J., Bulirsch, R.: Introduction to Numerical Analysis. Springer, New York (1980)
Vapnik, V.N.: Estimation of Dependencies Based on Empirical Data. Nauka, Moscow (1979). Translated into English: Springer, New York, (1982)
Vapnik, V.N.: The Nature of Statistical Learning Theory. Springer, New York (1995)
von Mises, R.: Mathematical Theory of Probability and Statistics. Academic Press, New York (1964)
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
Copyright information
© 2010 Springer-Verlag Berlin Heidelberg
About this chapter
Cite this chapter
Bezruchko, B.P., Smirnov, D.A. (2010). Restoration of Explicit Temporal Dependencies. In: Extracting Knowledge From Time Series. Springer Series in Synergetics. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-12601-7_7
Download citation
DOI: https://doi.org/10.1007/978-3-642-12601-7_7
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-12600-0
Online ISBN: 978-3-642-12601-7
eBook Packages: Physics and AstronomyPhysics and Astronomy (R0)