Numerical Algorithms

, Volume 47, Issue 2, pp 109–131 | Cite as

Fast generalized cross validation using Krylov subspace methods

  • Roger B. Sidje
  • Alan B. Williams
  • Kevin Burrage
Original Paper


The task of fitting smoothing spline surfaces to meteorological data such as temperature or rainfall observations is computationally intensive. The generalized cross validation (GCV) smoothing algorithm, if implemented using direct matrix techniques, is O(n 3) computationally, and memory requirements are O(n 2). Thus, for data sets larger than a few hundred observations, the algorithm is prohibitively slow. The core of the algorithm consists of solving series of shifted linear systems, and iterative techniques have been used to lower the computational complexity and facilitate implementation on a variety of supercomputer architectures. For large data sets though, the execution time is still quite high. In this paper we describe a Lanczos based approach that avoids explicitly solving the linear systems and dramatically reduces the amount of time required to fit surfaces to sets of data.


Lanczos Linear systems Generalized cross validation 

Mathematics Subject Classifications (2000)

65F10 65D10 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Ashby, S., Manteuffel, T., Saylor, P.: A taxonomy for conjugate gradient methods. SIAM J. Numer. Anal. 27, 1542–1568 (1990)MATHCrossRefMathSciNetGoogle Scholar
  2. 2.
    Axelsson, O.: A class of iterative methods for finite element equations. Comput. Methods Appl. Mech. Eng. 9, 123–137 (1976)CrossRefMathSciNetGoogle Scholar
  3. 3.
    Barrett, R., Berry, M., Chan, T.F., Demmel, J., Donato, J., Dongarra, J., Eijkhout, V., Pozo, R., Romine, C., Van der Vorst, H.: Templates for the Solution of Linear Systems: Building Blocks for Iterative Methods, 2nd edn. SIAM, Philadelphia, PA, USA (1994)Google Scholar
  4. 4.
    Burrage, K., Erhel, J., Pohl, B., Williams, A.: A deflation technique for systems of linear equations. SIAM J. Sci. Comput. 19(4), 1245–1260 (1998)MATHCrossRefMathSciNetGoogle Scholar
  5. 5.
    Burrage, K., Williams, A., Erhel, J., Pohl, B.: Implementation of a generalised cross validation algorithm using deflation techniques for systems of linear equations. Appl. Numer. Math. 19, 17–31 (1995)MATHCrossRefMathSciNetGoogle Scholar
  6. 6.
    Choquet, R.: Étude de la methode de Newton-GMRES. Application aux équations de Navier–Stokes compressibles. Ph.D. thesis, University of Rennes 1. No 1499, Dec 1995Google Scholar
  7. 7.
    Concus, P., Golub, G., Meurant, G.: Block preconditioning for the conjugate gradient method. SIAM J. Sci. Statist. Comp. 6, 220–252 (1985)MATHCrossRefMathSciNetGoogle Scholar
  8. 8.
    Datta, B.N., Saad, Y.: Arnoldi methods for large Sylvester-like observer matrix equations, and an associated algorithm ofr partial spectrum assignment. Linear Algebra Appl. 154–156, 225–244 (1991)CrossRefMathSciNetGoogle Scholar
  9. 9.
    Eisenstat, S.C.: Efficient implementation of a class of preconditioned conjugate gradient methods. SIAM J. Sci. Statist. Comput. 2, 1–4 (1981)MATHCrossRefMathSciNetGoogle Scholar
  10. 10.
    Erhel, J., Burrage, K., Pohl, B.: Restarted GMRES preconditioned by deflation. J. Comput. Appl. Math. 69, 303–318 (1996)MATHCrossRefMathSciNetGoogle Scholar
  11. 11.
    Freund, R.W.: Solution of shifted linear systems by quasi-minimal residual iterations. In: Reichel, L., Ruttan, A., Varga, R.S. (eds.) Numerical Linear Algebra, pp. 101–121. de Gruyter, Berlin (1993)Google Scholar
  12. 12.
    Gear, C.W., Saad, Y.: Iterative solution of linear equations in ODE codes. SIAM J. Sci. Statist. Comput. 4(4), 583–601 (1983)MATHCrossRefMathSciNetGoogle Scholar
  13. 13.
    Golub, G., Heath, M., Wahba, G.: Generalized cross validation as a method for choosing a good ridge parameter. Technometrics 21, 215–224 (1979)MATHCrossRefMathSciNetGoogle Scholar
  14. 14.
    Golub, G.H., Van Loan, C.F.: Matrix Computations, 3rd edn. The Johns Hopkins University Press, Baltimore (1996)MATHGoogle Scholar
  15. 15.
    Golub, G., Meurant, G.: Matrices, moments and quadrature. In: Griffiths, D.F., Waston, G.A. (eds.) Numerical Analysis 93. Pitman Research Notes in Mathematics 303, Longman Scientific and Technical (1993)Google Scholar
  16. 16.
    Golub, G.H., von Matt, U.: Generalized cross-validation for large scale problems. J. Comput. Graph. Stat. 6(1), 1–34 (1997)CrossRefGoogle Scholar
  17. 17.
    Gu, C., Bates, D., Chen, Z., Wahba, G.: The computation of GCV functions through Householder tridiagonalisation with application to the fitting of interaction spline models. SIAM J. Matrix Anal. Appl. 10, 459–480 (1989)CrossRefMathSciNetGoogle Scholar
  18. 18.
    Hestenes, M., Stiefel, E.: Methods of conjugate gradients for solving linear systems. NBS J. Research 49, 409–436 (1952)MATHMathSciNetGoogle Scholar
  19. 19.
    Hutchinson, M.F.: A stochastic estimator of the trace of the influence matrix for Laplacian smoothing splines. Commun. Stat. Simul. Comput. 18, 1059–1076 (1989)MATHCrossRefMathSciNetGoogle Scholar
  20. 20.
    Hutchinson, M.F., DeHoog, F.: Smoothing noisy data with spline functions. Numer. Math. 47, 99–106 (1985)MATHCrossRefMathSciNetGoogle Scholar
  21. 21.
    Johnson, O., Michelli, C., Paul, G.: Polynomial preconditioners for conjugate gradient calculations. SIAM J. Numer. Anal. 20, 362–376 (1983)MATHCrossRefMathSciNetGoogle Scholar
  22. 22.
    Meijerink, J.A., van der Vorst, H.A.: An iterative solution method for linear systems of which the coefficient matrix is a symmetric M-matrix. Math. Comput. 31, 148–152 (1977)MATHCrossRefGoogle Scholar
  23. 23.
    Meurant, G.: A review of the inverse of tridiagonal and block tridiagonal matrices. SIAM J. Matrix Anal. Appl. 13(3), 707–728 (1992)MATHCrossRefMathSciNetGoogle Scholar
  24. 24.
    O’Leary, D.P.: The block conjugate gradient algorithm and related methods. Linear Algebra Appl. 29, 293–322 (1980)MATHCrossRefMathSciNetGoogle Scholar
  25. 25.
    Ortega, J.: Introduction to Parallel and Vector Solution of Linear Systems. Plenum Press (1988)Google Scholar
  26. 26.
    Parlett, B.: The Symmetric Eigenvalue Problem. Prentice Hall (1980)Google Scholar
  27. 27.
    Parlett, B.N.: A new look at the Lanczos method for solving symmetric systems of linear equations. Linear Algebra Appl. 29, 323–346 (1980)MATHCrossRefMathSciNetGoogle Scholar
  28. 28.
    Parlett, B.N., Scott, D.S.: The Lanczos algorithm with selective orthogonalization. Math. Comput. 33, 217–238 (1979)MATHCrossRefMathSciNetGoogle Scholar
  29. 29.
    Philippe, B., Sidje, R.B.: Cumulative solutions of Markov processes by Krylov subspaces. In: Sixth International Conference of Scientific Computing, Benin City, Nigeria, 24–28 Jan 1992Google Scholar
  30. 30.
    Saad, Y.: A flexible inner outer preconditioned GMRES algorithm. SIAM J. Sci. Comput. 14, 461–469 (1993)MATHCrossRefMathSciNetGoogle Scholar
  31. 31.
    Saad, Y., Schultz, M.H.: GMRES: a generalized minimal residual algorithm for solving nonsymmetric linear systems. SIAM J. Sci. Statist. Comput. 7, 856–869 (1986)MATHCrossRefMathSciNetGoogle Scholar
  32. 32.
    Schoenauer, W., Weiss, R.: An engineering approach to generalized conjugate gradient methods and beyond. Appl. Numer. Math. 19, 175–206 (1995)MATHCrossRefMathSciNetGoogle Scholar
  33. 33.
    Sidje, R.B.: Parallel algorithms for large sparse matrix exponentials: application to numerical transient analysis of Markov processes. Ph.D. thesis, University of Rennes 1, July 1994Google Scholar
  34. 34.
    Simon, H.D.: The Lanczos algorithm with partial reorthogonalization. Math. Comput. 42, 115–142 (1984)MATHCrossRefGoogle Scholar
  35. 35.
    van der Vorst, H.: An iterative solution method for solving f(A)x = b using Krylov subspace information obtained for the symmetric positive definite matrix A. J. Comput. Appl. Math. 18, 249–263 (1987)MATHCrossRefMathSciNetGoogle Scholar
  36. 36.
    Wahba, G.: Smoothing noisy data by spline functions. Numer. Math. 24, 383–393 (1975)MATHCrossRefMathSciNetGoogle Scholar
  37. 37.
    Wahba, G.: Spline bases, regularization, and generalized cross validation for solving approximation problems with large quantities of noisy data. In: Cheney, W. (ed.) Approximation Theory III, pp. 905–912. Academic Press, New York (1980)Google Scholar
  38. 38.
    Wahba, G.: A comparison of GCV and GML for choosing the smoothing parameter in the generalized spline smoothing problem. Ann. Statist. 13, 1378–1402 (1985)MATHCrossRefMathSciNetGoogle Scholar
  39. 39.
    Wahba, G.: Spline Models for Observational Data. SIAM. CBMS-NSF Regional Conference Series in Applied Mathematics, vol. 59 (1990)Google Scholar
  40. 40.
    Wahba, G., Wendelberger, J.: Some new mathematical methods for variational objective analysis using splines and cross validation. Mon. Weather Rev. 108, 1122–1145 (1980)CrossRefGoogle Scholar
  41. 41.
    Wilkinson, J.: The Algebraic Eigenvalue Problem. Claredon Press (1965)Google Scholar
  42. 42.
    Williams, A., Burrage, K.: Surface fitting using GCV smoothing splines on supercomputers. Conference proceedings, Supercomputing95, San Diego, USA (1995)Google Scholar

Copyright information

© Springer Science+Business Media, LLC. 2007

Authors and Affiliations

  • Roger B. Sidje
    • 1
  • Alan B. Williams
    • 2
  • Kevin Burrage
    • 1
  1. 1.Advanced Computational Modelling Centre, Department of MathematicsThe University of QueenslandBrisbaneAustralia
  2. 2.Sandia National Labs.AlbuquerqueNew Mexico

Personalised recommendations