Gradient Estimation Schemes for Noisy Functions
- 144 Downloads
In this paper, we analyze different schemes for obtaining gradient estimates when the underlying functions are noisy. Good gradient estimation is important e.g. for nonlinear programming solvers. As error criterion, we take the norm of the difference between the real and estimated gradients. The total error can be split into a deterministic error and a stochastic error. For three finite-difference schemes and two design of experiments (DoE) schemes, we analyze both the deterministic errors and stochastic errors. We derive also optimal stepsizes for each scheme, such that the total error is minimized. Some of the schemes have the nice property that this stepsize minimizes also the variance of the error. Based on these results, we show that, to obtain good gradient estimates for noisy functions, it is worthwhile to use DoE schemes. We recommend to implement such schemes in NLP solvers.
KeywordsDesign of experiments finite differences gradient estimatation noisy functions
Unable to display preview. Download preview PDF.
- 1.Griewank, A. 1989On Automatic DifferentiationIri, M.Tanabe, K. eds. Mathematical Programming.KTK Scientific PublishersTokyo, Japan83107Google Scholar
- 2.Dixon, L.C.W. 1994On Automatic Differentiation and Continuous OptimizationNATO Advanced Study Institutes Series.434501512Google Scholar
- 3.Kiefer, J., Wolfowitz, J. 1952Stochastic Estimation of a Regression FunctionAnnals of Mathematical Statistics.23462466Google Scholar
- 4.Blum, J.R. 1954Multidimensional Stochastic Approximation MethodsAnnals of Mathematical Statistics.25737744Google Scholar
- 5.Ermoliev, Y. 1980Stochastic Quasigradient MethodsErmoliev, Y.Wets, R.J.B. eds. Numerical Techniques for Stochastic Optimization.Springer-VerlagBerlin, GermanyChapter 6Google Scholar
- 6.Donohue, J.M., Houck, E.C., Myers, R.H. 1993Simulation Designs for Controlling Second-Order Bias in First-Order Response SurfacesOperations Research.41880902Google Scholar
- 7.Donohue, J.M., Houck, E.C., Myers, R.H. 1995Simulation Designs for the Estimation of Quadratic Response Surface Gradients in the Presence of Model MisspecificationManagement Science.41244262Google Scholar
- 10.Montgomery, D.C. 1984Design and Analysis of Experiments2WileyNew York, NYGoogle Scholar
- 11.Box, G.E.P., Hunter, W.G., Hunter, J.S. 1987Statistics for ExperimentersWileyNew York, NYGoogle Scholar
- 12.Gill, P.E., Murray, W., Wright, M.H. 1981Practical OptimizationAcademic PressLondon, UKGoogle Scholar