On Penalized Least-Squares: Its Mean Squared Error and a Quasi-Optimal Weight Ratio

  • Burkhard Schaffrin

It is well known in a Random Effects Model, that the Best inhomogeneously LInear Prediction (inhomBLIP) of the random effects vector is equivalently generated by the standard Least-Squares (LS) approach. This LS solution is based on an objective function that consists of two parts, the first related to the observations and the second to the prior information on the random effects; for more details, we refer to the book by Rao, Toutenburg, Shalabh and Heumann (2008). We emphasize that, in this context, the second part cannot be interpreted as “penalization term”.

A very similar objective function, however, could be applied in the Gauss-Markov model where no prior information is available for the unknown parameters. In this case, the additional term would serve as “penalization” indeed as it forces the Penalized Least-Squares (PLS) solution into a chosen neighborhood, not specialized through the model. This idea goes, at least, back to Tykhonov (1963) and Phillips (1962) and has since become known as (a special case of) “Tykhonov regularization” for which the weight ratio between the first and the second term in the objective function determines the degree of smoothing to which the estimated parameters are subjected to. This weight ratio is widely known as “Tykhonov regularization parameter”; for more details, we refer to Grafarend and Schaffrin (1993) or Engl et al. (1996), for instance.


Regularization Parameter Bayesian Estimate LInear Prediction Error Matrix Similar Objective Function 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Baksalary JK, Kala R (1983) Partial ordering between matrices one of which is of rank one. Bulletin of Polish Academy of Sciences: Mathematics 31: 5-7MATHMathSciNetGoogle Scholar
  2. Engl H, Hanke M, Neubauer A (1996) Regularization of Inverse Problems. Kluwer: Dordrecht/NLMATHGoogle Scholar
  3. Goldberger AS (1962) Best linear unbiased prediction in the genelarized linear regression model. Journal of American Statistical Association. 57: 369-375MATHCrossRefMathSciNetGoogle Scholar
  4. Golub GH, Heath M, Wahba G (1979) Generalized cross-validation as a method for choosing a good ridge parameter. Technometrics 21: 215-223MATHCrossRefMathSciNetGoogle Scholar
  5. Grafarend E, Schaffrin B (1993) Adjustment Computations in Linear Models (in German), Bibliograph. Institute MannheimGoogle Scholar
  6. Hansen PC, O’Leary DP (1993) The use of the L-curve in the regularization of discrete ill-posed problems, SIAM Journal of Science and Computation. 14: 1487-1503MATHCrossRefMathSciNetGoogle Scholar
  7. Marshall AW, Olkin I (1979) Inequalities. Theory of Majorization and its Applications, Academic Press, New YorkGoogle Scholar
  8. Phillips DL (1962) A technique for the numerical solution of certain integral equations of the first kind, Journal of the Association for Computing Machinery. 9: 84-96MATHMathSciNetGoogle Scholar
  9. Rao CR (1976) Estimation of parameters in a linear model, Annals of Statistics. 4: 1023-1037MATHCrossRefMathSciNetGoogle Scholar
  10. Rao CR, Kleffe J (1988) Estimation of Variance Components and Applications, North Holland: Amsterdam/NLMATHGoogle Scholar
  11. Rao CR, Toutenburg H, Shalabh, Heumann C (2008) Linear Models and Generalizations. Least Squares and Alternatives (3rd edition) Springer, Berlin Heidelberg New YorkGoogle Scholar
  12. Schaffrin B (1983) Estimation of variance-covariance components for heterogenous replicated measurements (in German), German Geo-detic Community, Publication C-282, Munich/GermanyGoogle Scholar
  13. Schaffrin B (1985) The geodetic datum with stochastic prior informa-tion (in German), German Geodetic Community, Publication C-313, Munich/ GermanyGoogle Scholar
  14. Schaffrin B (1995) A comparison of inverse techniques: Regularization, weight estimation and homBLUP, IUGG General Assembly, IAG Scientific Meeting U7, Boulder/COGoogle Scholar
  15. Schaffrin B (2005) On the optimal choice of the regularization parameter through variance ratio estimation, 14th International Workshop on Matrices and Statistics, Auckland/NZGoogle Scholar
  16. Searle SR, Casella G, McCulloch CE (1992) Variance Components, Wiley, New YorkMATHCrossRefGoogle Scholar
  17. Tykhonov AN (1963) The regularization of incorrectly posed problems, Soviet Mathematics Doklady 4: 1624-1627Google Scholar

Copyright information

© Physica-Verlag Heidelberg 2008

Authors and Affiliations

  • Burkhard Schaffrin
    • 1
    • 2
  1. 1.School of Earth SciencesOhio State UniversityColumbusUSA
  2. 2.Geodetic InstituteUniversity of KarlsruheGermany

Personalised recommendations