Skip to main content
SpringerLink
Log in

Diagonal Bundle Method for Nonsmooth Sparse Optimization

  • Published:
Journal of Optimization Theory and Applications Aims and scope Submit manuscript

Abstract

We propose an efficient diagonal bundle method for sparse nonsmooth, possibly nonconvex optimization. The convergence of the proposed method is proved for locally Lipschitz continuous functions, which are not necessary differentiable or convex. The numerical experiments have been made using problems with up to million variables. The results to be presented confirm the usability of the diagonal bundle method especially for extremely large-scale problems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Outrata, J., Kočvara, M., Zowe, J.: Nonsmooth Approach to Optimization Problems with Equilibrium Constraints. Theory, Applications and Numerical Results. Kluwert Academic Publisher, Dordrecht (1998)

    Book  MATH  Google Scholar 

  2. Moreau, J.J., Panagiotopoulos, P.D., Strang, G. (eds.): Topics in Nonsmooth Mechanics. Birkhäuser Verlag, Basel (1988)

    MATH  Google Scholar 

  3. Mistakidis, E.S., Stavroulakis, G.E.: Nonconvex Optimization in Mechanics. Smooth and Nonsmooth Algorithms, Heuristics and Engineering Applications by the F.E.M. Kluwert Academic Publishers, Dordrecht (1998)

    Google Scholar 

  4. Clarke, F.H., Ledyaev, Y.S., Stern, R.J., Wolenski, P.R.: Nonsmooth Analysis and Control Theory. Springer, New York (1998)

    MATH  Google Scholar 

  5. Haslinger, J., Neittaanmäki, P.: Finite Element Approximation for Optimal Shape, Material and Topology Design, 2nd edn. John Wiley & Sons, Chichester (1996)

    MATH  Google Scholar 

  6. Kärkkäinen, T., Heikkola, E.: Robust formulations for training multilayer perceptrons. Neural. Comput. 16, 837–862 (2004)

    Article  MATH  Google Scholar 

  7. Äyrämö, S.: Knowledge mining using robust clustering. Ph.D. thesis, University of Jyväskylä, Department of Mathematical Information Technology (2006).

  8. Bradley, P.S., Fayyad, U.M., Mangasarian, O.L.: Mathematical programming for data mining: formulations and challenges. INFORMS J. Comput. 11, 217–238 (1999)

    Article  MathSciNet  MATH  Google Scholar 

  9. Demyanov, V.F., Bagirov, A.M., Rubinov, A.M.: A method of truncated codifferential with application to some problems of cluster analysis. J. Glob. Optim. 23(1), 63–80 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  10. Astorino, A., Fuduli, A.: Nonsmooth optimization techniques for semi-supervised classification. IEEE Trans. Pattern. Anal. Mach. Intell. 29(12), 2135–2142 (2007)

    Article  Google Scholar 

  11. Astorino, A., Fuduli, A., Gorgone, E.: Nonsmoothness in classification problems. Optim. Methods Softw. 23(5), 675–688 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  12. Bergeron, C., Moore, G., Zaretzki, J., Breneman, C., Bennett, K.: Fast bundle algorithm for multiple instance learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(6), 1068–1079 (2012).

  13. Carrizosa, E., Morales, D.R.: Supervised classification and mathematical optimization. Comput. Oper. Res. 40(1), 150–165 (2013)

    Article  MathSciNet  Google Scholar 

  14. Lemaréchal, C.: Nondifferentiable optimization. In: G.L. Nemhauser, A.H.G. Rinnooy Kan, M.J. Todd (eds.) Optimization, pp. 529–572. Elsevier North-Holland Inc, New York (1989).

  15. Beck, A., Teboulle, M.: Mirror descent and nonlinear projected subgradient methods for convex optimization. Oper. Res. Lett. 31(3), 167–175 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  16. Ben-Tal, A., Nemirovski, A.: Non-Euclidean restricted memory level method for large-scale convex optimization. Math. Program. 102(3), 407–456 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  17. Shor, N.Z.: Minimization Methods for Non-Differentiable Functions. Springer-Verlag, Berlin (1985)

    Book  MATH  Google Scholar 

  18. Uryasev, S.P.: Algorithms for nondifferentiable optimization problems. J. Optim. Theory Appl. 71, 359–388 (1991)

    Article  MathSciNet  Google Scholar 

  19. Gaudioso, M., Monaco, M.F.: Variants to the cutting plane approach for convex nondifferentiable optimization. Optimization 25, 65–75 (1992)

    Article  MathSciNet  MATH  Google Scholar 

  20. Hiriart-Urruty, J.B., Lemaréchal, C.: Convex Analysis and Minimization Algorithms II. Springer-Verlag, Berlin (1993)

    MATH  Google Scholar 

  21. Kiwiel, K.C.: Methods of Descent for Nondifferentiable Optimization. Lecture Notes in Mathematics 1133. Springer-Verlag, Berlin (1985).

  22. Lukšan, L., Vlček, J.: Globally convergent variable metric method for convex nonsmooth unconstrained minimization. J. Optim. Theory Appl. 102(3), 593–613 (1999)

    Article  MathSciNet  MATH  Google Scholar 

  23. Mäkelä, M.M.: Survey of bundle methods for nonsmooth optimization. Optim. Methods Softw. 17(1), 1–29 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  24. Mäkelä, M.M., Neittaanmäki, P.: Nonsmooth Optimization: Analysis and Algorithms with Applications to Optimal Control. World Scientific Publishing Co., Singapore (1992)

    Book  MATH  Google Scholar 

  25. Sagastizábal, C., Solodov, M.: An infeasible bundle method for nonsmooth convex constrained optimization without a penalty function or a filter. SIAM J. Optim. 16(1), 146–169 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  26. Schramm, H., Zowe, J.: A version of the bundle idea for minimizing a nonsmooth function: conceptual idea, convergence analysis, numerical results. SIAM J. Optim. 2(1), 121–152 (1992)

    Article  MathSciNet  MATH  Google Scholar 

  27. Polak, E., Royset, J.O.: Algorithms for finite and semi-finite min-max-min problems using adaptive smoothing techniques. J. Optim. Theory Appl. 119, 421–457 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  28. Burke, J.V., Lewis, A.S., Overton, M.L.: A robust gradient sampling algorithm for nonsmooth, nonconvex optimization. SIAM J. Optim. 15, 751–779 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  29. Apkarian, P., Noll, D., Prot, O.: A trust region spectral bundle method for non-convex eigenvalue optimization. SIAM J. Optim. 19(1), 281–306 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  30. Fuduli, A., Gaudioso, M., Nurminski, E.A.: A splitting bundle approach for non-smooth non-convex minimization. Optimization, (2013, in press ), doi:10.1080/02331934.2013.840625.

  31. Fuduli, A., Gaudioso, M., Giallombardo, G.: Minimizing nonconvex nonsmooth functions via cutting planes and proximity control. SIAM J. Optim. 14(3), 743–756 (2004)

    Article  MathSciNet  MATH  Google Scholar 

  32. Hare, W., Sagastizábal, C.: A redistributed proximal bundle method for nonconvex optimization. SIAM J. Optim. 20(5), 2442–2473 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  33. Karmitsa, N., Tanaka Filho, M., Herskovits, J.: Globally convergent cutting plane method for nonconvex nonsmooth minimization. J. Optim. Theory Appl. 148(3), 528–549 (2011)

  34. Noll, D., Prot, O., Rondepierre, A.: A proximity control algorithm to minimize nonsmooth and nonconvex functions. Pac. J. Optim. 4(3), 571–604 (2008)

    MathSciNet  MATH  Google Scholar 

  35. Haarala, M.: Large-scale nonsmooth optimization: Variable metric bundle method with limited memory. Ph.D. thesis, University of Jyväskylä, Department of Mathematical Information Technology (2004).

  36. Haarala, M., Miettinen, K., Mäkelä, M.M.: New limited memory bundle method for large-scale nonsmooth optimization. Optim. Methods Softw. 19(6), 673–692 (2004)

    Article  MathSciNet  MATH  Google Scholar 

  37. Haarala, N., Miettinen, K., Mäkelä, M.M.: Globally convergent limited memory bundle method for large-scale nonsmooth optimization. Math. Program. 109(1), 181–205 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  38. Bagirov, A.M., Karmitsa, N., Mäkelä, M.M.: Introduction to Nonsmooth Optimization: Theory, Practice and Software. Springer International Publishing (2014).

  39. Karmitsa, N., Bagirov, A., Mäkelä, M.M.: Comparing different nonsmooth optimization methods and software. Optim. Methods Softw. 27(1), 131–153 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  40. Herskovits, J., Goulart, E.: Sparse quasi-Newton matrices for large scale nonlinear optimization. In: Proceedings of the 6th Word Congress on Structurla and Multidisciplinary Optimization (2005).

  41. Clarke, F.H.: Optimization and Nonsmooth Analysis. Wiley-Interscience, New York (1983)

    MATH  Google Scholar 

  42. Karmitsa, N., Mäkelä, M.M., Ali, M.M.: Limited memory interior point bundle method for large inequality constrained nonsmooth minimization. Appl. Math. Comput. 198(1), 382–400 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  43. Vlček, J., Lukšan, L.: Globally convergent variable metric method for nonconvex nondifferentiable unconstrained minimization. J. Optim. Theory Appl. 111(2), 407–430 (2001)

    Article  MathSciNet  MATH  Google Scholar 

  44. Byrd, R.H., Nocedal, J., Schnabel, R.B.: Representations of quasi-Newton matrices and their use in limited memory methods. Math. Program. 63, 129–156 (1994)

    Article  MathSciNet  MATH  Google Scholar 

  45. Lemaréchal, C., Strodiot, J.J., Bihain, A.: On a bundle algorithm for nonsmooth optimization. In: Mangasarian, O.L., Mayer, R.R., Robinson, S.M. (eds.) Nonlinear Programming, pp. 245–281. Academic Press, New York (1981)

    Google Scholar 

  46. Mifflin, R.: A modification and an extension of Lemaréchal’s algorithm for nonsmooth minimization. Mat. Program. Study 17, 77–90 (1982)

    Article  MathSciNet  MATH  Google Scholar 

  47. Bihain, A.: Optimization of upper semidifferentiable functions. J. Optim. Theory Appl. 4, 545–568 (1984)

    Article  MathSciNet  Google Scholar 

  48. Fletcher, R.: Practical Methods of Optimization, 2nd edn. Wiley, Chichester (1987)

    MATH  Google Scholar 

  49. Nocedal, J.: Updating Quasi-Newton matrices with limited storage. Math. Comput. 35(151), 773–782 (1980)

    Article  MathSciNet  MATH  Google Scholar 

  50. Fletcher, R.: An optimal positive definite update for sparse Hessian matrices. SIAM J. Optim. 5(1), 192–218 (1995)

    Article  MathSciNet  MATH  Google Scholar 

  51. Fletcher, R., Grothey, A., Leyffer, S.: Computing sparse Hessian and Jacobian approximations with optimal hereditary properties. University of Dundee Numerical Analysis Report NA/164 (1995).

  52. Toint, P.L.: On sparse and symmetric matrix updating subject to a linear equation. Math. Comput. 31(140), 954–961 (1977)

    Article  MathSciNet  MATH  Google Scholar 

  53. Mäkelä, M.M.: Multiobjective proximal bundle method for nonconvex nonsmooth optimization: Fortran subroutine MPBNGC 2.0. Reports of the Department of Mathematical Information Technology, Series B. Scientific Computing, B. 13/2003 University of Jyväskylä, Jyväskylä (2003).

  54. Bagirov, A.M., Ganjehlou, A.N.: A secant method for nonsmooth optimization. Submitted (2009).

  55. Bagirov, A.M., Ganjehlou, A.N.: A quasisecant method for minimizing nonsmooth functions. Optim. Methods Softw. 25(1), 3–18 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  56. Lukšan, L.: Dual method for solving a special problem of quadratic programming as a subproblem at linearly constrained nonlinear minimax approximation. Kybernetika 20, 445–457 (1984)

    MathSciNet  MATH  Google Scholar 

  57. Miettinen, K., Mäkelä, M.M.: Synchronous approach in interactive multiobjective optimization. Eur. J. Oper. Res. 170(3), 909–922 (2006)

    Article  MATH  Google Scholar 

Download references

Acknowledgments

The work was financially supported by the University of Turku (Finland) and Magnus Ehrnrooth foundation.

Author information

Authors and Affiliations

  1. Department of Mathematics and Statistics, University of Turku, 20014, Turku, Finland

    Napsu Karmitsa

Authors
  1. Napsu Karmitsa
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Napsu Karmitsa.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Karmitsa, N. Diagonal Bundle Method for Nonsmooth Sparse Optimization. J Optim Theory Appl 166, 889–905 (2015). https://doi.org/10.1007/s10957-014-0666-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10957-014-0666-8

Keywords

Mathematics Subject Classification

Search

Navigation