Springer Nature is making SARS-CoV-2 and COVID-19 research free. View research | View latest news | Sign up for updates

The principle of penalized empirical risk in severely ill-posed problems


We study a standard method of regularization by projections of the linear inverse problem Y=Ax+ε, where ε is a white Gaussian noise, and A is a known compact operator. It is assumed that the eigenvalues of AA* converge to zero with exponential decay. Such behavior of the spectrum is typical for inverse problems related to elliptic differential operators. As model example we consider recovering of unknown boundary conditions in the Dirichlet problem for the Laplace equation on the unit disk. By using the singular value decomposition of A, we construct a projection estimator of x. The bandwidth of this estimator is chosen by a data-driven procedure based on the principle of minimization of penalized empirical risk. We provide non–asymptotic upper bounds for the mean square risk of this method and we show, in particular, that this approach gives asymptotically minimax estimators in our model example.

This is a preview of subscription content, log in to check access.


  1. 1.

    Akaike, H.: Information theory and an extension of the maximum likelihood principle. Proc. 2nd Intern. Symp. Inf. Theory, Petrov P.N. and Csaki F. (eds.), Budapest, 1973, pp. 267–281

  2. 2.

    Barron, A., Birge, L., Massart, P.: Risk bounds for model selection via penalization. Probab. Theory Relat. Fields 113, 301–413 (1999)

  3. 3.

    Belitser, E.N., Levit, Ya, B.: On minimax filtering on ellipsoids. Math. Meth. Statist. 4, 259–273 (1995)

  4. 4.

    Cavalier, L., Golubev, G., Lepski, O., Tsybakov, A: (2003) Block thresholding and sharp adaptive estimation in severely ill-posed inverse problems. Probab. Theory Appl. 3 to appear.

  5. 5.

    Cavalier, L., Tsybakov, A.: Sharp adaptation for inverse problems with random noise. Probab. Theory Relat. Fields 123, 323–354 (2002)

  6. 6.

    Cavalier, L., Golubev, G.K., Picard, D., Tsybakov, A.B.: Oracle inequalities for inverse problems. Ann. Stat. 30 (3), 843–874 (2002)

  7. 7.

    Engl, H., Hanke, M., Nuebauer, A.: Regularization of inverse problems. Kluwer academic publishers, 2000

  8. 8.

    Golubev, G.K., Khasminski, R.Z.: Statistical approach to some inverse boundary problems for partial differential equations. Probl. Inform. Transm. 35 (2), 51–66 (1999)

  9. 9.

    Efromovich, S.: Robust and efficient recovery of a signal passed through a filter and then contaminated by non-Gaussian noise. IEEE Trans. Inform. Theory 43, 1184–1191 (1997)

  10. 10.

    Ermakov, M.S.: On optimal solutions of the deconvolution problem. Inverse Probl. 6 (5), 863–872 (1990)

  11. 11.

    Johnstone, I.M.: Wavelet shrinkage for correlated data and inverse problems: adaptivity results. Statistica Sinica 9, 51–83 (1999)

  12. 12.

    Johnstone, I.M., Silverman, B.W.: Wavelet threshold estimators for data with correlated noise. J. Royal Stat. Soc. Ser. B. 59, 319–351 (1997)

  13. 13.

    Kneip, A.: Ordered linear smoothers. Ann. Stat. 22, 835–866 (1994)

  14. 14.

    Landau, H.J., Pollak, H.Q.: Prolate spheroidal wave function, Fourier analysis, and uncertainty – II. Bell System Tech. J. 40, 65–84 (1961)

  15. 15.

    Landau, H.J., Pollak, H.Q.: Prolate spheroidal wave function, Fourier analysis, and uncertainty – III. Bell System Tech. J. 41, 1295–1336 (1962)

  16. 16.

    Lavrentiev, M.M.: Some improperly posed problems of mathematical physics. Springer Verlag, Berlin Heidelberg New-York, 1967

  17. 17.

    Mair, B., Ruymgaart, F.H.: Statistical estimation in Hilbert scale. SIAM J. Appl. Math. 56, 1424–1444 (1996)

  18. 18.

    Mallows, C.L.: Some comments on C p . Technometrics 15, 661–675 (1973)

  19. 19.

    Pinsker, M.S.: Optimal filtering of square integrable signals in Gaussian white noise. Probl. Inform. Transm. 16, 120–133 (1980)

  20. 20.

    Shibata, R.: An optimal selection of regression variables. Boimetrika 68, 45–54 (1981)

  21. 21.

    Slepjan, D., Pollak, H.Q.: Prolate spheroidal wave function, Fourier analysis, and uncertainty – I. Bell System Tech. J. 40, 43–64 (1961)

  22. 22.

    Slepjan, D.: Some asymptotic expansions for prolate spheroidal wave functions. J. Math. Rhys. 44, 99–140 (1965)

  23. 23.

    Stein, C.M.: Estimation of the mean of a multivariate normal distribution. Ann. Stat. 9, 1135–1151 (1981)

  24. 24.

    Sudakov, V.N., Khalfin, L.A.: Statistical approach to ill-posed problems in mathematical physics. Soviet Mathematics Doklady 157, 1094–1096 (1964)

  25. 25.

    Sullivan, F.O’.: A statistical perspective on ill-posed inverse problems. Stat. Sci. 1, 502–527 (1996)

  26. 26.

    Tsybakov, A.B.: On the best rate of adaptive estimation in some inverse problems. C.R. Acad. Sci. Paris 330 Série I, 835–840 (2000)

Download references

Author information

Correspondence to Yu Golubev.

Additional information

Mathematics Subject Classification (2000):Primary 62G05, 62G20; secondary 62C20

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Golubev, Y. The principle of penalized empirical risk in severely ill-posed problems. Probab. Theory Relat. Fields 130, 18–38 (2004).

Download citation


  • Ill-posed problem
  • Partial differential equation
  • Singular value decomposition
  • Projection estimator
  • Minimax risk
  • Penalization
  • Empirical risk