Skip to main content
SpringerLink
Log in

Projection algorithms for nonconvex minimization with application to sparse principal component analysis

  • Published:
Journal of Global Optimization Aims and scope Submit manuscript

Abstract

We consider concave minimization problems over nonconvex sets. Optimization problems with this structure arise in sparse principal component analysis. We analyze both a gradient projection algorithm and an approximate Newton algorithm where the Hessian approximation is a multiple of the identity. Convergence results are established. In numerical experiments arising in sparse principal component analysis, it is seen that the performance of the gradient projection algorithm is very similar to that of the truncated power method and the generalized power method. In some cases, the approximate Newton algorithm with a Barzilai–Borwein Hessian approximation and a nonmonotone line search can be substantially faster than the other algorithms, and can converge to a better solution.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Alizadeh, A.A., et al.: Distinct types of diffuse large b-cell lymphoma identified by gene expression profiling. Nature 403, 503–511 (2000)

    Article  Google Scholar 

  2. Barzilai, J., Borwein, J.M.: Two point step size gradient methods. IMA J. Numer. Anal. 8, 141–148 (1988)

    Article  MathSciNet  MATH  Google Scholar 

  3. Beck, A., Teboulle, M.: Mirror descent and nonlinear projected subgradient methods for convex optimization. Oper. Res. Lett. 31, 167–175 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  4. Bertsekas, D.P.: Projected Newton methods for optimization problems with simple constraints. SIAM J. Control Optim. 20, 221–246 (1982)

    Article  MathSciNet  MATH  Google Scholar 

  5. Bhaskara, A., Charikar, M., Chlamtac, E., Feige, U., Vijayaraghavan, A.: Detecting high log-densities: an \(o (n 1/4)\) approximation for densest k-subgraph. In: Proceedings of the Forty-Second ACM Symposium on Theory of Computing, ACM, pp. 201–210 (2010)

  6. Boldi, P., Rosa, M., Santini, M., Vigna, S.: Layered label propagation: A multiresolution coordinate-free ordering for compressing social networks. In: Proceedings of the 20th International Conference on World Wide Web, ACM Press (2011)

  7. Boldi, P., Vigna, S.: The WebGraph framework I: Compression techniques. In: Proceedings of the Thirteenth International World Wide Web Conference (WWW 2004), Manhattan, USA, ACM Press, pp. 595–601 (2004)

  8. Cadima, J., Jolliffe, I.T.: Loading and correlations in the interpretation of principle compenents. J. Appl. Stat. 22, 203–214 (1995)

    Article  MathSciNet  Google Scholar 

  9. Candes, E., Wakin, M.: An introduction to compressive sampling. IEEE Signal Process. Mag. 25, 21–30 (2008)

    Article  Google Scholar 

  10. Chen, S.S., Donoho, D.L., Saunders, M.A.: Atomic decomposition by basis pursuit. SIAM J. Sci. Comput. 20, 33–61 (1998)

    Article  MathSciNet  MATH  Google Scholar 

  11. Clarkson, K.L.: Coresets, sparse greedy approximation, and the Frank-Wolfe algorithm. ACM Trans. Algorithms (TALG) 6, 63 (2010)

    MathSciNet  MATH  Google Scholar 

  12. d’Aspremont, A., Bach, F., Ghaoui, L.E.: Optimal solutions for sparse principal component analysis. J. Mach. Learn. Res. 9, 1269–1294 (2008)

    MathSciNet  MATH  Google Scholar 

  13. Donoho, D.L.: Compressed sensing. IEEE Trans. Inform. Theory 52, 1289–1306 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  14. Efron, B., Hastie, T., Johnstone, I., Tibshirani, R.: Least angle regression. Ann. Stat. 32, 407–499 (2004)

    Article  MathSciNet  MATH  Google Scholar 

  15. Frank, M., Wolfe, P.: An algorithm for quadratic programming. Nav. Res. Logist. Q 3, 95–110 (1956)

    Article  MathSciNet  Google Scholar 

  16. Grippo, L., Lampariello, F., Lucidi, S.: A nonmonotone line search technique for Newton’s method. SIAM J. Numer. Anal. 23, 707–716 (1986)

    Article  MathSciNet  MATH  Google Scholar 

  17. Hager, W.W., Zhang, H.: A new active set algorithm for box constrained optimization. SIAM J. Optim. 17, 526–557 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  18. Hazan, E., Kale, S.: Projection-free online learning, In: Langford, J., Pineau, J. (eds.) Proceedings of the 29th International Conference on Machine Learning, Omnipress, pp. 521–528 (2012)

  19. Jaggi, M.: Revisiting Frank-Wolfe: Projection-free sparse convex optimization. In: Dasgupta, S., McAllester, D. (eds.) Proceedings of the 30th International Conference on Machine Learning, vol. 28, pp. 427–435 (2013)

  20. Jeffers, J.: Two case studies in the application of principal components. Appl. Stat. 16, 225–236 (1967)

    Article  Google Scholar 

  21. Jenatton, R., Obozinski, G., Bach, F.: Structured sparse principal component analysis. In: International Conference on Artificial Intelligence and Statistics (AISTATS) (2010)

  22. Jolliffe, I.T., Trendafilov, N.T., Uddin, M.: A modified principal component technique based on the LASSO. J. Comput. Graph. Stat. 12, 531–547 (2003)

    Article  MathSciNet  Google Scholar 

  23. Journée, M., Nesterov, Y., Richtárik, P., Sepulchre, R.: Generalized power method for sparse principal component analysis. J. Mach. Learn. Res. 11, 517–553 (2010)

    MathSciNet  MATH  Google Scholar 

  24. Khuller, S., Saha, B.: On finding dense subgraphs, In: Albers, S., Marchetti-Spaccamela, A., Matias, Y., Nikoletseas, S., Thomas, W. (eds.) Automata, Languages and Programming, pp. 597–608. Springer, New York (2009)

  25. Lacoste-Julien, S., Jaggi, M., Schmidt, M., Pletscher, P.: Block-coordinate Frank-Wolfe optimization for structural SVMs. In: Dasgupta, S., McAllester, D. (eds.) Proceedings of the 30th International Conference on Machine Learning, vol. 28, pp. 53–61 (2013)

  26. Luss, R., Teboulle, M.: Convex approximations to sparse PCA via Lagrangian duality. Oper. Res. Lett. 39, 57–61 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  27. Luss, R., Teboulle, M.: Conditional gradient algorithms for rank-one matrix approximations with a sparsity constraint. SIAM Rev. 55, 65–98 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  28. Ramaswamy, S., Tamayo, P., Rifkin, R., Mukherjee, S., Yeang, C.-H., Angelo, M., Ladd, C., Reich, M., Latulippe, E., Mesirov, J.P., et al.: Multiclass cancer diagnosis using tumor gene expression signatures. Proc. Natl. Acad. Sci. USA 98, 15149–15154 (2001)

    Article  Google Scholar 

  29. Rockafellar, R.T.: Convex Analysis. Princeton University Press, Princeton (1970)

    Book  MATH  Google Scholar 

  30. Sriperumbudur, B.K., Torres, D.A., Lanckriet, G.R.: A majorization-minimization approach to the sparse generalized eigenvalue problem. Mach. Learn. 85, 3–39 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  31. Takeda, A., Niranjan, M., Gotoh, J.-Y., Kawahara, Y.: Simultaneous pursuit of out-of-sample performance and sparsity in index tracking portfolios. Comput. Manag. Sci. 10, 21–49 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  32. van den Berg, E., Friedlander, M.P.: Probing the pareto frontier for basis pursuit solutions. SIAM J. Sci. Comput. 31, 890–912 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  33. Wright, S.J., Nowak, R.D., Figueiredo, M.A.T.: Sparse reconstruction by separable approximation. IEEE Trans. Signal Process. 57, 2479–2493 (2009)

    Article  MathSciNet  Google Scholar 

  34. Ye, Y., Zhang, J.: Approximation of dense-n/2-subgraph and the complement of min-bisection. J. Glob. Optim. 25, 55–73 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  35. Yuan, X.-T., Zhang, T.: Truncated power method for sparse eigenvalue problems. J. Mach. Learn. Res. 14, 899–925 (2013)

    MathSciNet  MATH  Google Scholar 

  36. Zou, H., Hastie, T., Tibshirani, R.: Sparse principal component analysis. J. Comput. Graph. Stat. 15, 265–286 (2006)

    Article  MathSciNet  Google Scholar 

Download references

Author information

Author notes

  1. Jiajie Zhu

    Present address: Computer Science Department, Boston College, Chestnut Hill, MA, USA

Authors and Affiliations

  1. Department of Mathematics, University of Florida, PO Box 118105, Gainesville, FL, 32611-8105, USA

    William W. Hager & Jiajie Zhu

  2. IBM T. J. Watson Research Center, Yorktown Heights, NY, 20598, USA

    Dzung T. Phan

Authors
  1. William W. Hager
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dzung T. Phan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jiajie Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to William W. Hager.

Additional information

The authors gratefully acknowledge support by the National Science Foundation under Grants 1115568 and 1522629 and by the Office of Naval Research under Grants N00014-11-1-0068 and N00014-15-1-2048.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hager, W.W., Phan, D.T. & Zhu, J. Projection algorithms for nonconvex minimization with application to sparse principal component analysis. J Glob Optim 65, 657–676 (2016). https://doi.org/10.1007/s10898-016-0402-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10898-016-0402-z

Keywords

Search

Navigation