Skip to main content
SpringerLink
Log in

Fast generalized cross validation using Krylov subspace methods

  • Original Paper
  • Published:
Numerical Algorithms Aims and scope Submit manuscript

Abstract

The task of fitting smoothing spline surfaces to meteorological data such as temperature or rainfall observations is computationally intensive. The generalized cross validation (GCV) smoothing algorithm, if implemented using direct matrix techniques, is O(n 3) computationally, and memory requirements are O(n 2). Thus, for data sets larger than a few hundred observations, the algorithm is prohibitively slow. The core of the algorithm consists of solving series of shifted linear systems, and iterative techniques have been used to lower the computational complexity and facilitate implementation on a variety of supercomputer architectures. For large data sets though, the execution time is still quite high. In this paper we describe a Lanczos based approach that avoids explicitly solving the linear systems and dramatically reduces the amount of time required to fit surfaces to sets of data.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Ashby, S., Manteuffel, T., Saylor, P.: A taxonomy for conjugate gradient methods. SIAM J. Numer. Anal. 27, 1542–1568 (1990)

    Article  MATH  MathSciNet  Google Scholar 

  2. Axelsson, O.: A class of iterative methods for finite element equations. Comput. Methods Appl. Mech. Eng. 9, 123–137 (1976)

    Article  MathSciNet  Google Scholar 

  3. Barrett, R., Berry, M., Chan, T.F., Demmel, J., Donato, J., Dongarra, J., Eijkhout, V., Pozo, R., Romine, C., Van der Vorst, H.: Templates for the Solution of Linear Systems: Building Blocks for Iterative Methods, 2nd edn. SIAM, Philadelphia, PA, USA (1994)

    Google Scholar 

  4. Burrage, K., Erhel, J., Pohl, B., Williams, A.: A deflation technique for systems of linear equations. SIAM J. Sci. Comput. 19(4), 1245–1260 (1998)

    Article  MATH  MathSciNet  Google Scholar 

  5. Burrage, K., Williams, A., Erhel, J., Pohl, B.: Implementation of a generalised cross validation algorithm using deflation techniques for systems of linear equations. Appl. Numer. Math. 19, 17–31 (1995)

    Article  MATH  MathSciNet  Google Scholar 

  6. Choquet, R.: Étude de la methode de Newton-GMRES. Application aux équations de Navier–Stokes compressibles. Ph.D. thesis, University of Rennes 1. No 1499, Dec 1995

  7. Concus, P., Golub, G., Meurant, G.: Block preconditioning for the conjugate gradient method. SIAM J. Sci. Statist. Comp. 6, 220–252 (1985)

    Article  MATH  MathSciNet  Google Scholar 

  8. Datta, B.N., Saad, Y.: Arnoldi methods for large Sylvester-like observer matrix equations, and an associated algorithm ofr partial spectrum assignment. Linear Algebra Appl. 154–156, 225–244 (1991)

    Article  MathSciNet  Google Scholar 

  9. Eisenstat, S.C.: Efficient implementation of a class of preconditioned conjugate gradient methods. SIAM J. Sci. Statist. Comput. 2, 1–4 (1981)

    Article  MATH  MathSciNet  Google Scholar 

  10. Erhel, J., Burrage, K., Pohl, B.: Restarted GMRES preconditioned by deflation. J. Comput. Appl. Math. 69, 303–318 (1996)

    Article  MATH  MathSciNet  Google Scholar 

  11. Freund, R.W.: Solution of shifted linear systems by quasi-minimal residual iterations. In: Reichel, L., Ruttan, A., Varga, R.S. (eds.) Numerical Linear Algebra, pp. 101–121. de Gruyter, Berlin (1993)

    Google Scholar 

  12. Gear, C.W., Saad, Y.: Iterative solution of linear equations in ODE codes. SIAM J. Sci. Statist. Comput. 4(4), 583–601 (1983)

    Article  MATH  MathSciNet  Google Scholar 

  13. Golub, G., Heath, M., Wahba, G.: Generalized cross validation as a method for choosing a good ridge parameter. Technometrics 21, 215–224 (1979)

    Article  MATH  MathSciNet  Google Scholar 

  14. Golub, G.H., Van Loan, C.F.: Matrix Computations, 3rd edn. The Johns Hopkins University Press, Baltimore (1996)

    MATH  Google Scholar 

  15. Golub, G., Meurant, G.: Matrices, moments and quadrature. In: Griffiths, D.F., Waston, G.A. (eds.) Numerical Analysis 93. Pitman Research Notes in Mathematics 303, Longman Scientific and Technical (1993)

  16. Golub, G.H., von Matt, U.: Generalized cross-validation for large scale problems. J. Comput. Graph. Stat. 6(1), 1–34 (1997)

    Article  Google Scholar 

  17. Gu, C., Bates, D., Chen, Z., Wahba, G.: The computation of GCV functions through Householder tridiagonalisation with application to the fitting of interaction spline models. SIAM J. Matrix Anal. Appl. 10, 459–480 (1989)

    Article  MathSciNet  Google Scholar 

  18. Hestenes, M., Stiefel, E.: Methods of conjugate gradients for solving linear systems. NBS J. Research 49, 409–436 (1952)

    MATH  MathSciNet  Google Scholar 

  19. Hutchinson, M.F.: A stochastic estimator of the trace of the influence matrix for Laplacian smoothing splines. Commun. Stat. Simul. Comput. 18, 1059–1076 (1989)

    Article  MATH  MathSciNet  Google Scholar 

  20. Hutchinson, M.F., DeHoog, F.: Smoothing noisy data with spline functions. Numer. Math. 47, 99–106 (1985)

    Article  MATH  MathSciNet  Google Scholar 

  21. Johnson, O., Michelli, C., Paul, G.: Polynomial preconditioners for conjugate gradient calculations. SIAM J. Numer. Anal. 20, 362–376 (1983)

    Article  MATH  MathSciNet  Google Scholar 

  22. Meijerink, J.A., van der Vorst, H.A.: An iterative solution method for linear systems of which the coefficient matrix is a symmetric M-matrix. Math. Comput. 31, 148–152 (1977)

    Article  MATH  Google Scholar 

  23. Meurant, G.: A review of the inverse of tridiagonal and block tridiagonal matrices. SIAM J. Matrix Anal. Appl. 13(3), 707–728 (1992)

    Article  MATH  MathSciNet  Google Scholar 

  24. O’Leary, D.P.: The block conjugate gradient algorithm and related methods. Linear Algebra Appl. 29, 293–322 (1980)

    Article  MATH  MathSciNet  Google Scholar 

  25. Ortega, J.: Introduction to Parallel and Vector Solution of Linear Systems. Plenum Press (1988)

  26. Parlett, B.: The Symmetric Eigenvalue Problem. Prentice Hall (1980)

  27. Parlett, B.N.: A new look at the Lanczos method for solving symmetric systems of linear equations. Linear Algebra Appl. 29, 323–346 (1980)

    Article  MATH  MathSciNet  Google Scholar 

  28. Parlett, B.N., Scott, D.S.: The Lanczos algorithm with selective orthogonalization. Math. Comput. 33, 217–238 (1979)

    Article  MATH  MathSciNet  Google Scholar 

  29. Philippe, B., Sidje, R.B.: Cumulative solutions of Markov processes by Krylov subspaces. In: Sixth International Conference of Scientific Computing, Benin City, Nigeria, 24–28 Jan 1992

  30. Saad, Y.: A flexible inner outer preconditioned GMRES algorithm. SIAM J. Sci. Comput. 14, 461–469 (1993)

    Article  MATH  MathSciNet  Google Scholar 

  31. Saad, Y., Schultz, M.H.: GMRES: a generalized minimal residual algorithm for solving nonsymmetric linear systems. SIAM J. Sci. Statist. Comput. 7, 856–869 (1986)

    Article  MATH  MathSciNet  Google Scholar 

  32. Schoenauer, W., Weiss, R.: An engineering approach to generalized conjugate gradient methods and beyond. Appl. Numer. Math. 19, 175–206 (1995)

    Article  MATH  MathSciNet  Google Scholar 

  33. Sidje, R.B.: Parallel algorithms for large sparse matrix exponentials: application to numerical transient analysis of Markov processes. Ph.D. thesis, University of Rennes 1, July 1994

  34. Simon, H.D.: The Lanczos algorithm with partial reorthogonalization. Math. Comput. 42, 115–142 (1984)

    Article  MATH  Google Scholar 

  35. van der Vorst, H.: An iterative solution method for solving f(A)x = b using Krylov subspace information obtained for the symmetric positive definite matrix A. J. Comput. Appl. Math. 18, 249–263 (1987)

    Article  MATH  MathSciNet  Google Scholar 

  36. Wahba, G.: Smoothing noisy data by spline functions. Numer. Math. 24, 383–393 (1975)

    Article  MATH  MathSciNet  Google Scholar 

  37. Wahba, G.: Spline bases, regularization, and generalized cross validation for solving approximation problems with large quantities of noisy data. In: Cheney, W. (ed.) Approximation Theory III, pp. 905–912. Academic Press, New York (1980)

    Google Scholar 

  38. Wahba, G.: A comparison of GCV and GML for choosing the smoothing parameter in the generalized spline smoothing problem. Ann. Statist. 13, 1378–1402 (1985)

    Article  MATH  MathSciNet  Google Scholar 

  39. Wahba, G.: Spline Models for Observational Data. SIAM. CBMS-NSF Regional Conference Series in Applied Mathematics, vol. 59 (1990)

  40. Wahba, G., Wendelberger, J.: Some new mathematical methods for variational objective analysis using splines and cross validation. Mon. Weather Rev. 108, 1122–1145 (1980)

    Article  Google Scholar 

  41. Wilkinson, J.: The Algebraic Eigenvalue Problem. Claredon Press (1965)

  42. Williams, A., Burrage, K.: Surface fitting using GCV smoothing splines on supercomputers. Conference proceedings, Supercomputing95, San Diego, USA (1995)

Download references

Author information

Authors and Affiliations

  1. Advanced Computational Modelling Centre, Department of Mathematics, The University of Queensland, Brisbane, QLD, 4072, Australia

    Roger B. Sidje & Kevin Burrage

  2. Sandia National Labs., Albuquerque, NM, 87185, New Mexico

    Alan B. Williams

Authors
  1. Roger B. Sidje
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alan B. Williams
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kevin Burrage
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Roger B. Sidje.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Sidje, R.B., Williams, A.B. & Burrage, K. Fast generalized cross validation using Krylov subspace methods. Numer Algor 47, 109–131 (2008). https://doi.org/10.1007/s11075-007-9150-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11075-007-9150-y

Keywords

Mathematics Subject Classifications (2000)

Search

Navigation