Mathematical Programming

, Volume 175, Issue 1–2, pp 503–536 | Cite as

A parallelizable augmented Lagrangian method applied to large-scale non-convex-constrained optimization problems

  • Natashia Boland
  • Jeffrey Christiansen
  • Brian Dandurand
  • Andrew EberhardEmail author
  • Fabricio Oliveira
Full Length Paper Series A


We contribute improvements to a Lagrangian dual solution approach applied to large-scale optimization problems whose objective functions are convex, continuously differentiable and possibly nonlinear, while the non-relaxed constraint set is compact but not necessarily convex. Such problems arise, for example, in the split-variable deterministic reformulation of stochastic mixed-integer optimization problems. We adapt the augmented Lagrangian method framework to address the presence of nonconvexity in the non-relaxed constraint set and to enable efficient parallelization. The development of our approach is most naturally compared with the development of proximal bundle methods and especially with their use of serious step conditions. However, deviations from these developments allow for an improvement in efficiency with which parallelization can be utilized. Pivotal in our modification to the augmented Lagrangian method is an integration of the simplicial decomposition method and the nonlinear block Gauss–Seidel method. An adaptation of a serious step condition associated with proximal bundle methods allows for the approximation tolerance to be automatically adjusted. Under mild conditions optimal dual convergence is proven, and we report computational results on test instances from the stochastic optimization literature. We demonstrate improvement in parallel speedup over a baseline parallel approach.


Augmented Lagrangian method Proximal bundle method Nonlinear block Gauss–Seidel method Simplicial decomposition method Parallel computing 

Mathematics Subject Classification

90-08 90C06 90C11 90C15 90C25 90C26 90C30 90C46 

Supplementary material

Open image in new window
10107_2018_1253_MOESM1_ESM.pdf (88 kb)
Supplementary material 1 (pdf 87 KB)


  1. 1.
    Birge, J.R., Louveaux, F.: Introduction to Stochastic Programming. Springer, Berlin (2011)zbMATHCrossRefGoogle Scholar
  2. 2.
    Shor, N.: Minimization Methods for Non-differentiable Functions. Springer, New York (1985)zbMATHCrossRefGoogle Scholar
  3. 3.
    Bertsekas, D.: Nonlinear Programming. Athena Scientific, Belmont (1999)zbMATHGoogle Scholar
  4. 4.
    Ruszczyński, A.: Nonlinear Optimization. Princeton University Press, Princeton (2006)zbMATHCrossRefGoogle Scholar
  5. 5.
    Carøe, C.C., Schultz, R.: Dual decomposition in stochastic integer programming. Oper. Res. Lett. 24(1), 37–45 (1999)MathSciNetzbMATHCrossRefGoogle Scholar
  6. 6.
    Bertsekas, D.: Constrained Optimization and Lagrange Multiplier Methods. Academic Press, London (1982)zbMATHGoogle Scholar
  7. 7.
    Hestenes, M.R.: Multiplier and gradient methods. J. Optim. Theory Appl. 4, 303–320 (1969)MathSciNetzbMATHCrossRefGoogle Scholar
  8. 8.
    Powell, M.J.D.: A method for nonlinear constraints in minimization problems. In: Fletcher, R. (ed.) Optimization. Academic Press, New York (1969)Google Scholar
  9. 9.
    Rockafellar, R.: Monotone operators and the proximal point algorithm. SIAM J. Control Optim. 14(5), 877–898 (1976)MathSciNetzbMATHCrossRefGoogle Scholar
  10. 10.
    Lemaréchal, C.: An Extension of Davidon Methods to Non differentiable Problems, pp. 95–109. Springer, Berlin Heidelberg (1975)Google Scholar
  11. 11.
    de Oliveira, W., Sagastizábal, C.: Bundle methods in the XXIst century: a bird’s-eye view. Pesqui. Oper. 34, 647–670 (2014)CrossRefGoogle Scholar
  12. 12.
    Hare, W., Sagastizábal, C., Solodov, M.: A proximal bundle method for nonsmooth nonconvex functions with inexact information. Comput. Optim. Appl. 63, 1–28 (2016)MathSciNetzbMATHCrossRefGoogle Scholar
  13. 13.
    de Oliveira, W., Sagastizábal, C., Lemaréchal, C.: Convex proximal bundle methods in depth: a unified analysis for inexact oracles. Math. Program. 148(1), 241–277 (2014)MathSciNetzbMATHCrossRefGoogle Scholar
  14. 14.
    Amor, H.B., Desrosiers, J., Frangioni, A.: On the choice of explicit stabilizing terms in column generation. Discrete Appl. Math. 157(6), 1167–1184 (2009)MathSciNetzbMATHCrossRefGoogle Scholar
  15. 15.
    Bertsekas, D.: Incremental aggregated proximal and augmented Lagrangian algorithms. arXiv preprint arXiv:1509.09257 (2015)
  16. 16.
    Fischer, F., Helmberg, C.: A parallel bundle framework for asynchronous subspace optimization of nonsmooth convex functions. SIAM J. Optim. 24(2), 795–822 (2014)MathSciNetzbMATHCrossRefGoogle Scholar
  17. 17.
    Lubin, M., Martin, K., Petra, C., Sandıkçı, B.: On parallelizing dual decomposition in stochastic integer programming. Oper. Res. Lett. 41(3), 252–258 (2013)MathSciNetzbMATHCrossRefGoogle Scholar
  18. 18.
    Bertsekas, D.: Convex Optimization Algorithms. Athena Scientific, Belmont (2015)zbMATHGoogle Scholar
  19. 19.
    Holloway, C.: An extension of the Frank and Wolfe method of feasible directions. Math. Program. 6(1), 14–27 (1974)MathSciNetzbMATHCrossRefGoogle Scholar
  20. 20.
    Von Hohenbalken, B.: Simplicial decomposition in nonlinear programming algorithms. Math. Program. 13(1), 49–68 (1977)MathSciNetzbMATHCrossRefGoogle Scholar
  21. 21.
    Bonettini, S.: Inexact block coordinate descent methods with application to non-negative matrix factorization. IMA J. Numer. Anal. 31(4), 1431–1452 (2011)MathSciNetzbMATHCrossRefGoogle Scholar
  22. 22.
    Grippo, L., Sciandrone, M.: On the convergence of the block nonlinear Gauss–Seidel method under convex constraints. Oper. Res. Lett. 26(3), 127–136 (2000)MathSciNetzbMATHCrossRefGoogle Scholar
  23. 23.
    Hildreth, C.: A quadratic programming procedure. Nav. Res. Logist. Q. 4, 79–85 (1957). 361MathSciNetCrossRefGoogle Scholar
  24. 24.
    Tseng, P.: Convergence of a block coordinate descent method for nondifferentiable minimization. J. Optim. Theory Appl. 109, 475–494 (2001)MathSciNetzbMATHCrossRefGoogle Scholar
  25. 25.
    Warga, J.: Minimizing certain convex functions. SIAM J. Appl. Math. 11, 588–593 (1963)MathSciNetzbMATHCrossRefGoogle Scholar
  26. 26.
    Eckstein, J.: A practical general approximation criterion for methods of multipliers based on Bregman distances. Math. Program. 96(1), 61–86 (2003)MathSciNetzbMATHCrossRefGoogle Scholar
  27. 27.
    Eckstein, J., Silva, P.: A practical relative error criterion for augmented lagrangians. Math. Program. 141(1), 319–348 (2013)MathSciNetzbMATHCrossRefGoogle Scholar
  28. 28.
    Hamdi, A., Mahey, P., Dussault, J.P.: Recent Advances in Optimization: Proceedings of the 8th French–German Conference on Optimization Trier, July 21–26, 1996, chap. A New Decomposition Method in Nonconvex Programming via a Separable Augmented Lagrangian, pp. 90–104. Springer Berlin Heidelberg, Berlin, Heidelberg (1997)Google Scholar
  29. 29.
    Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn. 3(1), 1–122 (2011)zbMATHCrossRefGoogle Scholar
  30. 30.
    Gabay, D., Mercier, B.: A dual algorithm for the solution of nonlinear variational problems via finite element approximation. Comput. Math. Appl. 2, 17–40 (1976)zbMATHCrossRefGoogle Scholar
  31. 31.
    Glowinski, R., Marrocco, A.: Sur l’approximation, par elements finis d’ordre un, et la resolution, par penalisation-dualité, d’une classe de problems de dirichlet non lineares. Revue Française d’Automatique, Informatique, et Recherche Opérationelle 9, 41–76 (1975)zbMATHCrossRefGoogle Scholar
  32. 32.
    Chatzipanagiotis, N., Dentcheva, D., Zavlanos, M.: An augmented Lagrangian method for distributed optimization. Math. Program. 152(1), 405–434 (2014)MathSciNetzbMATHGoogle Scholar
  33. 33.
    Tappenden, R., Richtárik, P., Büke, B.: Separable approximations and decomposition methods for the augmented Lagrangian. Optim. Methods Softw. 30(3), 643–668 (2015)MathSciNetzbMATHCrossRefGoogle Scholar
  34. 34.
    Mulvey, J., Ruszczyński, A.: A diagonal quadratic approximation method for large scale linear programs. Oper. Res. Lett. 12(4), 205–215 (1992)MathSciNetzbMATHCrossRefGoogle Scholar
  35. 35.
    Ruszczyński, A.: On convergence of an augmented Lagrangian decomposition method for sparse convex optimization. Math. Oper. Res. 20(3), 634–656 (1995)MathSciNetzbMATHCrossRefGoogle Scholar
  36. 36.
    Kiwiel, K., Rosa, C., Ruszczyński, A.: Proximal decomposition via alternating linearization. SIAM J. Optim. 9(3), 668–689 (1999)MathSciNetzbMATHCrossRefGoogle Scholar
  37. 37.
    Lin, X., Pham, M., Ruszczyński, A.: Alternating linearization for structured regularization problems. J. Mach. Learn. Res. 15, 3447–3481 (2014)MathSciNetzbMATHGoogle Scholar
  38. 38.
    Chen, G., Teboulle, M.: A proximal-based decomposition method for convex minimization problems. Math. Program. 64, 81–101 (1994)MathSciNetzbMATHCrossRefGoogle Scholar
  39. 39.
    He, B., Liao, L.Z., Han, D., Yang, H.: A new inexact alternating directions method for monotone variational inequalities. Math. Program. 92, 103–118 (2002)MathSciNetzbMATHCrossRefGoogle Scholar
  40. 40.
    Boland, N., Christiansen, J., Dandurand, B., Eberhard, A., Linderoth, J., Luedtke, J., Oliveira, F.: Progressive hedging with a Frank–Wolfe based method for computing stochastic mixed-integer programming Lagrangian dual bounds. Optimization Online (2016).
  41. 41.
    Eckstein, J., Bertsekas, D.: On the Douglas–Rachford splitting method and the proximal point algorithm for maximal monotone operators. Math. Program. 55(1–3), 293–318 (1992)MathSciNetzbMATHCrossRefGoogle Scholar
  42. 42.
    Eckstein, J., Yao, W.: Understanding the Convergence of the Alternating Direction Method of Multipliers: Theoretical and Computational Perspectives. Rutgers University, New Brunswick (2014). Tech. repzbMATHGoogle Scholar
  43. 43.
    Mahey, P., Oualibouch, S., Tao, P.D.: Proximal decomposition on the graph of a maximal monotone operator. SIAM J. Optim. 5(2), 454–466 (1995)MathSciNetzbMATHCrossRefGoogle Scholar
  44. 44.
    Feizollahi, M.J., Costley, M., Ahmed, S., Grijalva, S.: Large-scale decentralized unit commitment. Int. J. Electr. Power Energy Syst. 73, 97–106 (2015)CrossRefGoogle Scholar
  45. 45.
    Gade, D., Hackebeil, G., Ryan, S.M., Watson, J.P., Wets, R.J.B., Woodruff, D.L.: Obtaining lower bounds from the progressive hedging algorithm for stochastic mixed-integer programs. Math. Program. 157(1), 47–67 (2016)MathSciNetzbMATHCrossRefGoogle Scholar
  46. 46.
    Rockafellar, R.T.: Augmented Lagrangians and applications of the proximal point algorithm in convex programming. Math. Oper. Res. 1(2), 97–116 (1976)MathSciNetzbMATHCrossRefGoogle Scholar
  47. 47.
    Rockafellar, R.: Convex Analysis. Princeton University Press, Princeton (1970)zbMATHCrossRefGoogle Scholar
  48. 48.
    Hathaway, R.J., Bezdek, J.C.: Grouped coordinate minimization using Newton’s method for inexact minimization in one vector coordinate. J. Optim. Theory Appl. 71(3), 503–516 (1991)MathSciNetzbMATHCrossRefGoogle Scholar
  49. 49.
    Kiwiel, K.C.: Approximations in proximal bundle methods and decomposition of convex programs. J. Optim. Theory Appl. 84(3), 529–548 (1995)MathSciNetzbMATHCrossRefGoogle Scholar
  50. 50.
    Bodur, M., Dash, S., Günlük, O., Luedtke, J.: Strengthened Benders cuts for stochastic integer programs with continuous recourse (2014). Last Accessed 13 Jan (2015)
  51. 51.
    Ahmed, S., Garcia, R., Kong, N., Ntaimo, L., Parija, G., Qiu, F., Sen, S.: SIPLIB: A stochastic integer programming test problem library (2015).
  52. 52.
    Ntaimo, L.: Decomposition algorithms for stochastic combinatorial optimization: Computational experiments and extensions. Ph.D. thesis (2004)Google Scholar
  53. 53.
    The MathWorks, Natick: MATLAB 2012b (2014)Google Scholar
  54. 54.
    IBM Corporation: IBM ILOG CPLEX Optimization Studio CPLEX Users Manual. Last Accessed 22 Aug (2016)
  55. 55.
    IBM Corporation: IBM ILOG CPLEX V12.5. Last Accessed 28 Jan (2016)
  56. 56.
    COmputational INfrastructure for Operations Research. Last Accessed 28 Jan (2016)
  57. 57.
    National Computing Infrastructure (NCI): NCI Website. Last Accessed 19 Nov 2016
  58. 58.
    Gertz, E., Wright, S.: Object-oriented software for quadratic programming. ACM Trans. Math. Softw. 29(1), 58–81 (2003)MathSciNetzbMATHCrossRefGoogle Scholar
  59. 59.
    Lubin, M., Petra, C., Anitescu, M., Zavala, V.: Scalable stochastic optimization of complex energy systems. In: Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 64:164:10. ACM, Seattle, WA (2011)Google Scholar
  60. 60.
    Clarke, F.: Optimization and Nonsmooth Analysis. Society for Industrial and Applied Mathematics (1990)Google Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature and Mathematical Optimization Society 2018

Authors and Affiliations

  • Natashia Boland
    • 1
  • Jeffrey Christiansen
    • 2
  • Brian Dandurand
    • 2
  • Andrew Eberhard
    • 2
    Email author
  • Fabricio Oliveira
    • 3
  1. 1.Georgia Institute of TechnologyAtlantaUSA
  2. 2.RMIT UniversityMelbourneAustralia
  3. 3.Aalto UniversityEspooFinland

Personalised recommendations