Abstract
Studies have shown that the surrogate subgradient method, to optimize non-smooth dual functions within the Lagrangian relaxation framework, can lead to significant computational improvements as compared to the subgradient method. The key idea is to obtain surrogate subgradient directions that form acute angles toward the optimal multipliers without fully minimizing the relaxed problem. The major difficulty of the method is its convergence, since the convergence proof and the practical implementation require the knowledge of the optimal dual value. Adaptive estimations of the optimal dual value may lead to divergence and the loss of the lower bound property for surrogate dual values. The main contribution of this paper is on the development of the surrogate Lagrangian relaxation method and its convergence proof to the optimal multipliers, without the knowledge of the optimal dual value and without fully optimizing the relaxed problem. Moreover, for practical implementations, a stepsizing formula that guarantees convergence without requiring the optimal dual value has been constructively developed. The key idea is to select stepsizes in a way that distances between Lagrange multipliers at consecutive iterations decrease, and as a result, Lagrange multipliers converge to a unique limit. At the same time, stepsizes are kept sufficiently large so that the algorithm does not terminate prematurely. At convergence, the lower-bound property of the surrogate dual is guaranteed. Testing results demonstrate that non-smooth dual functions can be efficiently optimized, and the new method leads to faster convergence as compared to other methods available for optimizing non-smooth dual functions, namely, the simple subgradient method, the subgradient-level method, and the incremental subgradient method.
Similar content being viewed by others
Notes
While Augmented Lagrangian relaxation has been a powerful method and can alleviate zigzagging, thereby reducing computational requirements, it is generally not used to optimize dual functions. Furthermore, the extra quadratic term makes the problem non-separable. Although methods were developed to overcome the resulting non-separability issue, they were not very effective.
Strictly speaking, when dealing with inequality constraints \(g(x)\le 0\), distances between multipliers and projections of multiples from the previous iteration are considered rather than distances between multipliers.
In the subgradient method, zero-subgradient implies that the optimum is obtained, and the algorithm terminates with the optimal primal solution. In the surrogate subgradient method, zero-surrogate subgradient implies that only a feasible solution is obtained, and the algorithm must proceed.
Initial stepsize \(c^{0}\) is initialized to be a positive scalar, therefore, stepsizes \(c^{k}\), \(k\) = 1, 2, ... satisfying (18) are positive.
When \(\beta ^k<<1\), the right-hand side of (35) decreases faster than the left-hand side as \(k\) increases. This leads to the contradiction, and the theorem is proved.
For the GAP15900 instance, the implementation of the new method may resemble that of the interleaved method [14] since only one subproblem is optimized at a time. The important difference between the new method and the interleaved method is the stepsizing formula.
References
Ermoliev, Y.M.: Methods for solving nonlinear extremal problems. Cybernetics 2(4), 1–17 (1966)
Polyak, B.T.: A general method of solving extremum problems. Sov. Math. Doklady 8, 593–597 (1967)
Polyak, B.T.: Minimization of unsmooth functionals. USSR Comput. Math. Math. Phys. 9(3), 14–29 (1969). (in Russian)
Shor, N.Z.: On the rate of convergence of the generalized gradient method. Cybernetics 4(3), 79–80 (1968)
Shor, N.Z.: Generalized gradient methods for non-smooth functions and their applications to mathematical programming problems. Econ. Math. Methods 12(2), 337–356 (1976). (in Russian)
Goffin, J.-L.: On the finite convergence of the relaxation method for solving systems of inequalities. Operations Research Center Report ORC 71–36, University of California at Berkeley, Berkeley (1971).
Goffin, J.-L., Kiwiel, K.: Convergence of a simple subgradient level method. Math. Program. 85(1), 207–211 (1999)
Nedic, A., Bertsekas, D.P.: Convergence rate of incremental subgradient algorithms. In: Uryasev, S., Pardalos, P.M. (eds.) Stochastic Optimization: Algorithms and Applications, pp. 263–304. Kluwer Academic, New York (2000)
Nedic, A., Bertsekas, D.: Incremental subgradient methods for nondifferentiable optimization. SIAM J. Optim. 56(1), 109–138 (2001)
Nesterov, Y.: Smooth minimization of non-smooth functions. Math. Program. 103(1), 127–152 (2005)
Bertsekas, D.P.: Incremental gradient, subgradient, and proximal methods for convex optimization: a survey. LIDS Technical Report no. 2848, MIT, (2010).
Bertsekas, D.P.: Incremental proximal methods for large scale convex optimization. Math. Program. 129, 163–195 (2011)
Lemarechal, C., Nemirovskii, A.S., Nesterov, Y.E.: New variants of bundle methods. Math. Program. 69, 111–147 (1995)
Kaskavelis, C.A., Caramanis, M.C.: Efficient Lagrangian relaxation algorithms for industry size job-shop scheduling problems. IIE Trans. 30(11), 1085–1097 (1998)
Zhao, X., Luh, P.B., Wang, J.: Surrogate gradient algorithm for Lagrangian relaxation. J. Optim. Theory Appl. 100(3), 699–712 (1999)
Luh, P.B., Blankson, W.E., Chen, Y., Yan, J.H., Stern, G.A., Chang, S.C., Zhao, F.: Payment cost minimization auction for the deregulated electricity markets using surrogate optimization. IEEE Trans. Power Syst. 21(2), 568–578 (2006)
Sun, T., Zhao, Q.C., Luh, P.B.: On the surrogate gradient algorithm for Lagrangian relaxation. J. Optim. Theory Appl. 133(3), 413–416 (2007)
Chang, T.S.: Comments on “Surrogate gradient algorithm for Lagrangian relaxation”. J. Optim. Theory Appl. 137(3), 691–697 (2008)
Bragin, M.A., Han, X., Luh, P.B., Yan, J.H.: Payment cost minimization using Lagrangian relaxation and modified surrogate optimization approach. In: Proceedings of the IEEE Power Engineering Society, General Meeting, Detroit, Michigan (2011)
Bragin, M.A., Luh, P.B., Yan, J.H., Yu, N., Han, X., Stern, G.A.: An efficient surrogate subgradient method within Lagrangian relaxation for the payment cost minimization problem. In: Proceedings of the IEEE Power Engineering Society, General Meeting, San Diego (2012)
Allen, E., Nelgason, R., Kennongton, J., Shettym, B.: A generalization of Polyak’s convergence result for subgradient optimization. Math. Program. 37(3), 309–317 (1987)
Bertsekas, D.P.: Nonlinear Programming. Athena Scientific, Massachusetts (2008)
Wah, B.W., Chen, Y.X.: Subgoal partitioning and global search for solving temporal planning problems in mixed space. Int. J. Artif. Intell. Tools 13(4), 767–790 (2004)
Chu, P.C., Beasley, J.E.: A genetic algorithm for the generalized assignment problem. Comput. Oper. Res. 24(1), 17–23 (1997)
Yagiura, M., Yamaguchi, T., Ibaraki, T.: A variable depth search algorithm with branching search for the generalized assignment problem. Optim. Methods Softw. 10, 419–441 (1998)
Yagiura, M., Ibaraki, T., Glover, F.: A path relinking approach with ejection chains for the generalized assignment problem. Eur. J. Oper. Res. 169(2), 548–569 (2006)
Avella, P., Boccia, M., Vasilyev, I.: A computational study of exact knapsack separation for the generalized assignment problem. Comput. Optim. Appl. 45(3), 543–555 (2010)
Posta, M., Ferland, J.A., Michelon, P.: An exact method with variable fixing for solving the generalized assignment problem. Comput. Optim. Appl. 52(3), 629–644 (2012)
Özbakir, L., Baykasoglu, A., Tapkan, P.: Bees algorithm for generalized assignment problem. Appl. Math. Comput. 215(11), 3782–3795 (2010)
Asahiro, Y., Ishibashi, M., Yamashita, M.: Independent and cooperative parallel search methods for the generalized assignment problem. Optim. Methods Softw. 18(2), 129–141 (2003)
Laguna, M., Kelly, J.P., Gonzalez-Velarde, J.L., Glover, F.: Tabu search for the multilevel generalized assignment problem. Eur. J. Oper. Res. 82, 176–189 (1995)
Koopmans, T.C., Beckmann, M.J.: Assignment problems and the location of economic activities. Econometrica 25(1), 53–76 (1957)
Dickey, J.W., Hopkins, J.W.: Campus building arrangement using TOPAZ. Transp. Res. 6, 59–68 (1972)
Elshafei, A.N.: Hospital layout as a quadratic assignment problem. Oper. Res. Q. 28, 167–179 (1977)
Geoffrion, A.M., Graves, G.W.: Scheduling parallel production lines with changeover costs: practical applications of a quadratic assignment/LP approach. Oper. Res. 24, 595–610 (1976)
Krarup, J., Pruzan, P.M.: Computer-aided layout design. Math. Program. Study 9, 75–94 (1978)
Burkard, R.E., Offermann, J.: Entwurf von Schreibmaschinentastaturen mittels quadratischer Zuordnungsprobleme. Zeitschrift fur Oper. Res. 21(4), 121–132 (1977). (in German)
Christofides, N., Benavent, E.: An exact algorithm for the quadratic assignment problem. Oper. Res. 37(5), 760–768 (1989)
Acknowledgments
This work was supported in part by grants from Southern California Edison and by the National Science Foundation under Grant ECCS–1028870. The authors would like to acknowledge Congcong Wang and Yaowen Yu for their careful perusal of the paper, insightful comments and valuable suggestions during numerous discussions.
Author information
Authors and Affiliations
Corresponding author
Additional information
Communicated by Fabián Flores-Bazàn.
Rights and permissions
About this article
Cite this article
Bragin, M.A., Luh, P.B., Yan, J.H. et al. Convergence of the Surrogate Lagrangian Relaxation Method. J Optim Theory Appl 164, 173–201 (2015). https://doi.org/10.1007/s10957-014-0561-3
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10957-014-0561-3
Keywords
- Non-smooth optimization
- Subgradient methods
- Surrogate subgradient method
- Lagrangian relaxation
- Mixed-integer programming