Journal of Optimization Theory and Applications

, Volume 126, Issue 3, pp 529–551 | Cite as

Gradient Estimation Schemes for Noisy Functions

  • R. C. M. Brekelmans
  • L. T. Driessen
  • H. J. M. Hamers
  • D. den. Hertog


In this paper, we analyze different schemes for obtaining gradient estimates when the underlying functions are noisy. Good gradient estimation is important e.g. for nonlinear programming solvers. As error criterion, we take the norm of the difference between the real and estimated gradients. The total error can be split into a deterministic error and a stochastic error. For three finite-difference schemes and two design of experiments (DoE) schemes, we analyze both the deterministic errors and stochastic errors. We derive also optimal stepsizes for each scheme, such that the total error is minimized. Some of the schemes have the nice property that this stepsize minimizes also the variance of the error. Based on these results, we show that, to obtain good gradient estimates for noisy functions, it is worthwhile to use DoE schemes. We recommend to implement such schemes in NLP solvers.


Design of experiments finite differences gradient estimatation noisy functions 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Griewank, A. 1989On Automatic DifferentiationIri, M.Tanabe, K. eds. Mathematical Programming.KTK Scientific PublishersTokyo, Japan83107Google Scholar
  2. 2.
    Dixon, L.C.W. 1994On Automatic Differentiation and Continuous OptimizationNATO Advanced Study Institutes Series.434501512Google Scholar
  3. 3.
    Kiefer, J., Wolfowitz, J. 1952Stochastic Estimation of a Regression FunctionAnnals of Mathematical Statistics.23462466Google Scholar
  4. 4.
    Blum, J.R. 1954Multidimensional Stochastic Approximation MethodsAnnals of Mathematical Statistics.25737744Google Scholar
  5. 5.
    Ermoliev, Y. 1980Stochastic Quasigradient MethodsErmoliev, Y.Wets, R.J.B. eds. Numerical Techniques for Stochastic Optimization.Springer-VerlagBerlin, GermanyChapter 6Google Scholar
  6. 6.
    Donohue, J.M., Houck, E.C., Myers, R.H. 1993Simulation Designs for Controlling Second-Order Bias in First-Order Response SurfacesOperations Research.41880902Google Scholar
  7. 7.
    Donohue, J.M., Houck, E.C., Myers, R.H. 1995Simulation Designs for the Estimation of Quadratic Response Surface Gradients in the Presence of Model MisspecificationManagement Science.41244262Google Scholar
  8. 8.
    Safizadeh, M.H. 2002Minimizing the Bias and Variance of the Gradient Estimate in RSM Simulation StudiesEuropean Journal of Operational Research.136121135CrossRefMathSciNetGoogle Scholar
  9. 9.
    Zazanis, M.A., Suri, R. 1993Convergence Rates of Finite-Difference Sensitivity Estimates for Stochastic SystemsOperations Research.41694703MathSciNetGoogle Scholar
  10. 10.
    Montgomery, D.C. 1984Design and Analysis of Experiments2WileyNew York, NYGoogle Scholar
  11. 11.
    Box, G.E.P., Hunter, W.G., Hunter, J.S. 1987Statistics for ExperimentersWileyNew York, NYGoogle Scholar
  12. 12.
    Gill, P.E., Murray, W., Wright, M.H. 1981Practical OptimizationAcademic PressLondon, UKGoogle Scholar

Copyright information

© Springer Science+Business Media, Inc. 2005

Authors and Affiliations

  • R. C. M. Brekelmans
    • 1
  • L. T. Driessen
    • 2
  • H. J. M. Hamers
    • 3
  • D. den. Hertog
    • 4
  1. 1.Staff Member, Center for Applied ResearchTilburg UniversityTilburgNetherlands
  2. 2.Staff Member, Center for Quantitative Methods BVTilburg UniversityEindhovenNetherlands
  3. 3.Associate Professor, Department of Econometrics and ORTilburg UniversityTilburgNetherlands
  4. 4.Professor, Department of Econometrics and ORTilburg UniversityTilburgNetherlands

Personalised recommendations