PDE acceleration: a convergence rate analysis and applications to obstacle problems

  • Jeff CalderEmail author
  • Anthony Yezzi


This paper provides a rigorous convergence rate and complexity analysis for a recently introduced framework, called PDE acceleration, for solving problems in the calculus of variations and explores applications to obstacle problems. PDE acceleration grew out of a variational interpretation of momentum methods, such as Nesterov’s accelerated gradient method and Polyak’s heavy ball method, that views acceleration methods as equations of motion for a generalized Lagrangian action. Its application to convex variational problems yields equations of motion in the form of a damped nonlinear wave equation rather than nonlinear diffusion arising from gradient descent. These accelerated PDEs can be efficiently solved with simple explicit finite difference schemes where acceleration is realized by an improvement in the CFL condition from \(\mathrm{d}t\sim \mathrm{d}x^2\) for diffusion equations to \(\mathrm{d}t\sim \mathrm{d}x\) for wave equations. In this paper, we prove a linear convergence rate for PDE acceleration for strongly convex problems, provide a complexity analysis of the discrete scheme, and show how to optimally select the damping parameter for linear problems. We then apply PDE acceleration to solve minimal surface obstacle problems, including double obstacles with forcing, and stochastic homogenization problems with obstacles, obtaining state-of-the-art computational results.


Accelerated gradient descent Nesterov acceleration Primal–dual methods Minimal surfaces Stochastic homogenization Nonlinear wave equations 

Mathematics Subject Classification

65M06 35Q93 65K10 49K20 



  1. 1.
    Armstrong, S., Hannukainen, A., Kuusi, T., Mourrat, J.C.: An iterative method for elliptic problems with rapidly oscillating coefficients. arXiv:1803.03551 (2018)
  2. 2.
    Armstrong, S., Kuusi, T., Mourrat, J.C.: Quantitative stochastic homogenization and large-scale regularity. arXiv:1705.05300 (2017)
  3. 3.
    Attouch, H., Goudou, X., Redont, P.: The heavy ball with friction method, I. The continuous dynamical system: global exploration of the local minima of a real-valued function by asymptotic analysis of a dissipative dynamical system. Commun. Contemp. Math. 2(01), 1–34 (2000)MathSciNetzbMATHGoogle Scholar
  4. 4.
    Badea, L., Tai, X.C., Wang, J.: Convergence rate analysis of a multiplicative Schwarz method for variational inequalities. SIAM J. Numer. Anal. 41(3), 1052–1073 (2003)MathSciNetzbMATHGoogle Scholar
  5. 5.
    Bähr, M., Breuß, M., Wunderlich, R.: Fast explicit diffusion for long-time integration of parabolic problems. In: AIP Conference Proceedings, vol. 1863, p. 410002. AIP Publishing (2017)Google Scholar
  6. 6.
    Ball, J.M.: The calculus of variations and materials science. Q. Appl. Math. 56(4), 719–740 (1998)MathSciNetzbMATHGoogle Scholar
  7. 7.
    Bardi, M., Capuzzo-Dolcetta, I.: Optimal Control and Viscosity Solutions of Hamilton–Jacobi–Bellman Equations. Springer, Berlin (2008)zbMATHGoogle Scholar
  8. 8.
    Benyamin, M., Calder, J., Sundaramoorthi, G., Yezzi, A.: Accelerated variational PDEs for efficient solution of regularized inversion problems. J. Math. Imaging Vis. (2019). Google Scholar
  9. 9.
    Bottou, L.: Large-scale machine learning with stochastic gradient descent. In: Proceedings of COMPSTAT’2010, pp. 177–186. Springer, Berlin (2010)Google Scholar
  10. 10.
    Braess, D., Carstensen, C., Hoppe, R.H.: Convergence analysis of a conforming adaptive finite element method for an obstacle problem. Numer. Math. 107(3), 455–471 (2007)MathSciNetzbMATHGoogle Scholar
  11. 11.
    Brezis, H., Sibony, M.: Méthodes d’approximation et d’itération pour les opérateurs monotones. Arch. Ration. Mech. Anal. 28(1), 59–82 (1968)zbMATHGoogle Scholar
  12. 12.
    Caffarelli, L.A.: The obstacle problem revisited. J. Fourier Anal. Appl. 4(4–5), 383–402 (1998)MathSciNetzbMATHGoogle Scholar
  13. 13.
    Chan, T.F., Vese, L.A.: Active contours without edges. IEEE Trans. Image Process. 10(2), 266–277 (2001)zbMATHGoogle Scholar
  14. 14.
    Crandall, M.G., Ishii, H., Lions, P.L.: User’s guide to viscosity solutions of second order partial differential equations. Bull. Am. Math. Soc. 27(1), 1–67 (1992)MathSciNetzbMATHGoogle Scholar
  15. 15.
    Evans, L.: Partial Differential Equations. Graduate Studies in Mathematics, vol. 19, 2nd edn. AMS, Providence (2002)Google Scholar
  16. 16.
    Goudou, X., Munier, J.: The gradient and heavy ball with friction dynamical systems: the quasiconvex case. Math. Program. 116(1–2), 173–191 (2009)MathSciNetzbMATHGoogle Scholar
  17. 17.
    Grewenig, S.: FSI schemes: fast semi-iterative solvers for PDEs and optimisation methods. In: Pattern Recognition: 38th German Conference, GCPR 2016, Hannover, Germany, September 12–15, 2016, Proceedings, vol. 9796, p. 91. Springer, Berlin (2016)Google Scholar
  18. 18.
    Hafner, D., Ochs, P., Weickert, J., Reißel, M., Grewenig, S.: FSI schemes: fast semi-iterative solvers for PDEs and optimisation methods. In: German Conference on Pattern Recognition, pp. 91–102. Springer, Berlin (2016)Google Scholar
  19. 19.
    Hintermüller, M., Ito, K., Kunisch, K.: The primal-dual active set strategy as a semismooth Newton method. SIAM J. Optim. 13(3), 865–888 (2002)MathSciNetzbMATHGoogle Scholar
  20. 20.
    Hintermüller, M., Kovtunenko, V.A., Kunisch, K.: Obstacle problems with cohesion: a hemivariational inequality approach and its efficient numerical solution. SIAM J. Optim. 21(2), 491–516 (2011)MathSciNetzbMATHGoogle Scholar
  21. 21.
    Hoppe, R.H.: Multigrid algorithms for variational inequalities. SIAM J. Numer. Anal. 24(5), 1046–1065 (1987)MathSciNetzbMATHGoogle Scholar
  22. 22.
    Johnson, C.: Adaptive finite element methods for the obstacle problem. Math. Models Methods Appl. Sci. 2(04), 483–487 (1992)MathSciNetzbMATHGoogle Scholar
  23. 23.
    LeVeque, R.J.: Finite Difference Methods for Ordinary and Partial Differential Equations: Steady-State and Time-Dependent Problems, vol. 98. SIAM, Philadelphia (2007)zbMATHGoogle Scholar
  24. 24.
    Lions, P.L., Mercier, B.: Splitting algorithms for the sum of two nonlinear operators. SIAM J. Numer. Anal. 16(6), 964–979 (1979)MathSciNetzbMATHGoogle Scholar
  25. 25.
    Majava, K., Tai, X.C.: A level set method for solving free boundary problems associated with obstacles. Int. J. Numer. Anal. Model 1(2), 157–171 (2004)MathSciNetzbMATHGoogle Scholar
  26. 26.
    Nesterov, Y.: A method of solving a convex programming problem with convergence rate o (1/k2). Soviet Math. Dokl. 27, 372–376 (1983)zbMATHGoogle Scholar
  27. 27.
    Polyak, B.T.: Some methods of speeding up the convergence of iteration methods. USSR Comput. Math. Math. Phys. 4(5), 1–17 (1964)Google Scholar
  28. 28.
    Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Physica D 60(1–4), 259–268 (1992)MathSciNetzbMATHGoogle Scholar
  29. 29.
    Schaeffer, H., Hou, T.Y.: An accelerated method for nonlinear elliptic pde. J. Sci. Comput. 69(2), 556–580 (2016)MathSciNetzbMATHGoogle Scholar
  30. 30.
    Scholz, R.: Numerical solution of the obstacle problem by the penalty method. Computing 32(4), 297–306 (1984)MathSciNetzbMATHGoogle Scholar
  31. 31.
    Schwarz, H.A.: Sur une définition erronée de l’aire d’une surface courbe. Gesammelte Math. Abh. 1, 309–311 (1890)Google Scholar
  32. 32.
    Su, W., Boyd, S., Candes, E.: A differential equation for modeling Nesterov’s accelerated gradient method: Theory and insights. In: Advances in Neural Information Processing Systems 27 (NIPS 2014), pp. 2510–2518 (2014) Google Scholar
  33. 33.
    Sundaramoorthi, G., Yezzi, A.: Accelerated optimization in the PDE framework: formulations for the manifold of diffeomorphisms. arXiv:1804.02307 (2018)
  34. 34.
    Sundaramoorthi, G., Yezzi, A.: Variational PDE’s for acceleration on manifolds and applications to diffeomorphisms. In: Advances in Neural Information Processing Systems 31 (NIPS 2018) (2018)Google Scholar
  35. 35.
    Sutskever, I., Martens, J., Dahl, G., Hinton, G.: On the importance of initialization and momentum in deep learning. In: International Conference on Machine Learning, pp. 1139–1147 (2013)Google Scholar
  36. 36.
    Tai, X.C.: Rate of convergence for some constraint decomposition methods for nonlinear variational inequalities. Numer. Math. 93(4), 755–786 (2003)MathSciNetzbMATHGoogle Scholar
  37. 37.
    Tai, X.C., Hahn, J., Chung, G.J.: A fast algorithm for Euler’s elastica model using augmented Lagrangian method. SIAM J. Imaging Sci. 4(1), 313–344 (2011)MathSciNetzbMATHGoogle Scholar
  38. 38.
    Tran, G., Schaeffer, H., Feldman, W.M., Osher, S.J.: An L1 penalty method for general obstacle problems. SIAM J. Appl. Math. 75(4), 1424–1444 (2015)MathSciNetzbMATHGoogle Scholar
  39. 39.
    Wang, F., Cheng, X.L.: An algorithm for solving the double obstacle problems. Appl. Math. Comput. 201(1–2), 221–228 (2008)MathSciNetzbMATHGoogle Scholar
  40. 40.
    Ward, C., Whitaker, N., Kevrekidis, I., Kevrekidis, P.: A toolkit for steady states of nonlinear wave equations: continuous time Nesterov and exponential time differencing schemes. arXiv:1710.05047 (2017)
  41. 41.
    Weickert, J., Grewenig, S., Schroers, C., Bruhn, A.: Cyclic schemes for PDE-based image analysis. Int. J. Comput. Vis. 118(3), 275–299 (2016)MathSciNetzbMATHGoogle Scholar
  42. 42.
    Wibisono, A., Wilson, A.C., Jordan, M.I.: A variational perspective on accelerated methods in optimization. Proc. Natl. Acad. Sci. 113(47), E7351–E7358 (2016)MathSciNetzbMATHGoogle Scholar
  43. 43.
    Woodard, R.P.: The theorem of Ostrogradsky. arXiv:1506.02210 (2015)
  44. 44.
    Yezzi, A., Sundaramoorthi, G.: Accelerated optimization in the PDE framework: formulations for the active contour case. arXiv:1711.09867 (2017)
  45. 45.
    You, Y.L., Kaveh, M.: Fourth-order partial differential equations for noise removal. IEEE Trans. Image Process. 9(10), 1723–1730 (2000)MathSciNetzbMATHGoogle Scholar
  46. 46.
    Zhang, Y.: Multilevel projection algorithm for solving obstacle problems. Comput. Math. Appl. 41(12), 1505–1513 (2001)MathSciNetzbMATHGoogle Scholar
  47. 47.
    Zhu, W., Chan, T.: Image denoising using mean curvature of image surface. SIAM J. Imaging Sci. 5(1), 1–32 (2012)MathSciNetzbMATHGoogle Scholar
  48. 48.
    Zosso, D., Osting, B.: A minimal surface criterion for graph partitioning. Inverse Probl. Imaging 10(4), 1149–1180 (2016)MathSciNetzbMATHGoogle Scholar
  49. 49.
    Zosso, D., Osting, B., Xia, M.M., Osher, S.J.: An efficient primal-dual method for the obstacle problem. J. Sci. Comput. 73(1), 416–437 (2017)MathSciNetzbMATHGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.School of MathematicsUniversity of MinnesotaMinneapolisUSA
  2. 2.School of Electrical and Computer EngineeringGeorgia Institute of TechnologyAtlantaUSA

Personalised recommendations