Reverse-Convex Programming for Sparse Image Codes

  • Matthias Heiler
  • Christoph Schnörr
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3757)


Reverse-convex programming (RCP) concerns global optimization of a specific class of non-convex optimization problems. We show that a recently proposed model for sparse non-negative matrix factorization (NMF) belongs to this class. Based on this result, we design two algorithms for sparse NMF that solve sequences of convex second-order cone programs (SOCP).

We work out some well-defined modifications of NMF that leave the original model invariant from the optimization viewpoint. They considerably generalize the sparse NMF setting to account for uncertainty in sparseness, for supervised learning, and, by dropping the non-negativity constraint, for sparsity-controlled PCA.


Positive Matrix Factorization Order Cone Sparsity Measure Sparse Image Project Gradient Descent 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Horst, R., Tuy, H.: Global Optimization. Springer, Berlin (1996)zbMATHGoogle Scholar
  2. 2.
    Hoyer, P.O.: Non-negative matrix factorization with sparseness constraints. J. of Mach. Learning Res. 5, 1457–1469 (2004)MathSciNetGoogle Scholar
  3. 3.
    Shen, J., Israël, G.W.: A receptor model using a specific non-negative transformation technique for ambient aerosol. Atmospheric Environment 23(10), 2289–2298 (1989)CrossRefGoogle Scholar
  4. 4.
    Paatero, P., Tapper, U.: Positive matrix factorization: A non-negative factor model with optimal utilization of error estimates of data values. Environmetrics 5, 111–126 (1994)CrossRefGoogle Scholar
  5. 5.
    Xu, W., Liu, X., Gong, Y.: Document clustering based on non-negative matrix factorization. In: SIGIR 2003: Proc. of the 26th Ann. Intl. ACM SIGIR Conf. on Res. and Developm. in Info. Retrieval, pp. 267–273. ACM Press, New York (2003)CrossRefGoogle Scholar
  6. 6.
    Hoyer, P.O., Hyvärinen, A.: A multi-layer sparse coding network learns contour coding from natural images. Vision Research 42(12), 1593–1605 (2002)CrossRefGoogle Scholar
  7. 7.
    Smaragdis, P., Brown, J.C.: Non-negative matrix factorization for polyphonic music transcription. In: IEEE Workshop on Appl. of Sign. Proc. to Audio and Acoustics, pp. 177–180 (2003)Google Scholar
  8. 8.
    Lee, D.D., Seung, H.S.: Learning the parts of objects by non-negative matrix factorization. Nature 401, 788–791 (1999)CrossRefGoogle Scholar
  9. 9.
    Donoho, D., Stodden, V.: When does non-negative matrix factorization give a correct decomposition into parts? In: Adv. in NIPS, vol. 17 (2004)Google Scholar
  10. 10.
    Li, S.Z., Hou, X.W., Zhang, H.J., Cheng, Q.S.: Learning spatially localized, parts-based representation. In: Proc. of CVPR (2001)Google Scholar
  11. 11.
    Wang, Y., Jia, Y., Hu, C., Turk, M.: Fisher non-negative matrix factorization for learning local features. In: Proc. Asian Conf. on Comp. Vision (2004)Google Scholar
  12. 12.
    Littlestone, N., Warmuth, M.: Relating data compression, learnability, and the Vapnik-Chervonenkis dimension. Tech. Rep., Univ. of Calif. Santa Cruz (1986)Google Scholar
  13. 13.
    Herbrich, R., Williamson, R.C.: Algorithmic luckiness. J. of Mach. Learning Res. 3, 175–212 (2002)MathSciNetCrossRefGoogle Scholar
  14. 14.
    Olshausen, B.A., Field, D.J.: Sparse coding with an overcomplete basis set: A strategy employed by V1? Vision Research 37, 3311–3325 (1997)CrossRefGoogle Scholar
  15. 15.
    Sturm, J.F.: Using SeDuMi 1.02, a Matlab toolbox for optimization over symmetric cones (updated version 1.05). Department of Econometrics, Tilburg University, Tilburg, The Netherlands (2001)Google Scholar
  16. 16.
    ApS, M. (ed.): The MOSEK optimization tools version 3.2 (Revision 8) User’s manual and reference, MOSEK ApS, Denmark (2005)Google Scholar
  17. 17.
    Lobo, M.S., Vandenberghe, L., Boyd, S., Lebret, H.: Applications of second-order cone programming. Linear Algebra and its Applications (1998)Google Scholar
  18. 18.
    Tuy, H.: Convex programs with an additional reverse convex constraint. J. of Optim. Theory and Applic. 52, 463–486 (1987)zbMATHMathSciNetCrossRefGoogle Scholar
  19. 19.
    Horst, R., Pardalos, P.M. (eds.): Handbook of Global Optimization. Kluwer Academic Publishers, Dordrecht (1995)zbMATHGoogle Scholar
  20. 20.
    d’Aspremont, A., Ghaoui, L.E., Jordan, M.I., Lanckriet, G.R.: A direct formulation for sparse PCA using semidefinite programming. In: Adv. in NIPS (2004)Google Scholar
  21. 21.
    Zou, H., Hastie, T., Tibshirani, R.: Sparse principal component analysis. J. of Comp. a. Graph. Statistics (to appear)Google Scholar
  22. 22.
    Chennubholta, C., Jepson, A.: Sparse PCA extracting multi-scale structure from data. In: Proc. of ICCV, pp. 641–647 (2001)Google Scholar
  23. 23.
    Paatero, P.: Least squares formulation of robust non-negative factor analysis. Chemometrics and Intelligent Laboratory Systems 37 (1997)Google Scholar
  24. 24.
    CBCL, CBCL face database #1. MIT Center For Biological and Computational Learning (2000),

Copyright information

© Springer-Verlag Berlin Heidelberg 2005

Authors and Affiliations

  • Matthias Heiler
    • 1
  • Christoph Schnörr
    • 1
  1. 1.Computer Vision, Graphics, and Pattern Recognition Group, Department of Mathematics and Computer ScienceUniversity of MannheimMannheimGermany

Personalised recommendations