Advertisement

Mathematical Programming

, Volume 169, Issue 1, pp 307–336 | Cite as

Minimization of transformed \(L_1\) penalty: theory, difference of convex function algorithm, and robust application in compressed sensing

  • Shuai Zhang
  • Jack Xin
Full Length Paper Series B

Abstract

We study the minimization problem of a non-convex sparsity promoting penalty function, the transformed \(l_1\) (TL1), and its application in compressed sensing (CS). The TL1 penalty interpolates \(l_0\) and \(l_1\) norms through a nonnegative parameter \(a \in (0,+\infty )\), similar to \(l_p\) with \(p \in (0,1]\), and is known to satisfy unbiasedness, sparsity and Lipschitz continuity properties. We first consider the constrained minimization problem, and discuss the exact recovery of \(l_0\) norm minimal solution based on the null space property (NSP). We then prove the stable recovery of \(l_0\) norm minimal solution if the sensing matrix A satisfies a restricted isometry property (RIP). We formulated a normalized problem to overcome the lack of scaling property of the TL1 penalty function. For a general sensing matrix A, we show that the support set of a local minimizer corresponds to linearly independent columns of A. Next, we present difference of convex algorithms for TL1 (DCATL1) in computing TL1-regularized constrained and unconstrained problems in CS. The DCATL1 algorithm involves outer and inner loops of iterations, one time matrix inversion, repeated shrinkage operations and matrix-vector multiplications. The inner loop concerns an \(l_1\) minimization problem on which we employ the Alternating Direction Method of Multipliers. For the unconstrained problem, we prove convergence of DCATL1 to a stationary point satisfying the first order optimality condition. In numerical experiments, we identify the optimal value \(a=1\), and compare DCATL1 with other CS algorithms on two classes of sensing matrices: Gaussian random matrices and over-sampled discrete cosine transform matrices (DCT). Among existing algorithms, the iterated reweighted least squares method based on \(l_{1/2}\) norm is the best in sparse recovery for Gaussian matrices, and the DCA algorithm based on \(l_1\) minus \(l_2\) penalty is the best for over-sampled DCT matrices. We find that for both classes of sensing matrices, the performance of DCATL1 algorithm (initiated with \(l_1\) minimization) always ranks near the top (if not the top), and is the most robust choice insensitive to the conditioning of the sensing matrix A. DCATL1 is also competitive in comparison with DCA on other non-convex penalty functions commonly used in statistics with two hyperparameters.

Keywords

Transformed \(l_1\) penalty Sparse signal recovery theory Difference of convex function algorithm Convergence analysis Coherent random matrices Compressed sensing Robust recovery 

Mathematics Subject Classification

90C26 65K10 90C90 

Notes

Acknowledgements

The authors would like to thank Professor Wenjiang Fu for referring us to [25], Professor Jong-Shi Pang for his helpful suggestions, and the anonymous referees for their constructive comments.

References

  1. 1.
    Ahn, M., Pang, J.-S., Xin, J.: Difference-of-convex learning: directional stationarity, optimality, and sparsity. SIAM J. Optim. 27(3), 1637–1665 (2017)MathSciNetCrossRefzbMATHGoogle Scholar
  2. 2.
    Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn. 3(1), 1–122 (2011)CrossRefzbMATHGoogle Scholar
  3. 3.
    Candès, E., Tao, T.: Decoding by linear programming. IEEE Trans. Inf. Theory 51(12), 4203–4215 (2005)MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    Candès, E., Rudelson, M., Tao, T., Vershynin, R.: Error correction via linear programming, In: 46th Annual IEEE Symposium on Foundations of Computer Science, pp. 668–681 (2005)Google Scholar
  5. 5.
    Candès, E., Romberg, J., Tao, T.: Robust uncertainty principles: exact signal reconstruction from highly incomplete Fourier information. IEEE Trans. Inf. Theory 52(2), 489–509 (2006)CrossRefzbMATHGoogle Scholar
  6. 6.
    Candès, E., Romberg, J., Tao, T.: Stable signal recovery from incomplete and inaccurate measurements. Commun. Pure Appl. Math. 59(8), 1207–1223 (2006)MathSciNetCrossRefzbMATHGoogle Scholar
  7. 7.
    Candès, E., Fernandez-Granda, C.: Super-resolution from noisy data. J. Fourier Anal. Appl. 19(6), 1229–1254 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  8. 8.
    Cao, W., Sun, J., Xu, Z.: Fast image deconvolution using closed-form thresholding formulas of regularization. J. Vis. Commun. Image Represent. 24(1), 31–41 (2013)CrossRefGoogle Scholar
  9. 9.
    Chartrand, R.: Nonconvex compressed sensing and error correction. ICASSP 3, III 889 (2007)Google Scholar
  10. 10.
    Chartrand, R., Yin, W.: Iteratively reweighted algorithms for compressive sensing. In: ICASSP pp. 3869–3872 (2008)Google Scholar
  11. 11.
    Cohen, A., Dahmen, W., DeVore, R.: Compressed sensing and the best k-term approximation. J. Am. Math. Soc. 22, 211–231 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  12. 12.
    Donoho, D.: Compressed sensing. IEEE Trans. Inf. Theory 52(4), 1289–1306 (2006)MathSciNetCrossRefzbMATHGoogle Scholar
  13. 13.
    Donoho, D., Elad, M.: Optimally sparse representation in general (nonorthogonal) dictionaries via \(\ell _1\) minimization. Proc. Natl. Acad. Sci. USA 100, 2197–2202 (2003)MathSciNetCrossRefzbMATHGoogle Scholar
  14. 14.
    Esser, E., Lou, Y., Xin, J.: A method for finding structured sparse solutions to non-negative least squares problems with applications. SIAM J. Imaging Sci. 6, 2010–2046 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  15. 15.
    Fan, J., Li, R.: Variable selection via nonconcave penalized likelihood and its oracle properties. J. Am. Stat. Assoc. 96(456), 1348–1360 (2001)MathSciNetCrossRefzbMATHGoogle Scholar
  16. 16.
    Fannjiang, A., Liao, W.: Coherence pattern–guided compressive sensing with unresolved grids. SIAM J. Imaging Sci. 5(1), 179–202 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  17. 17.
    Goldstein, T., Osher, S.: The split Bregman method for \(\ell _1\)-regularized problems. SIAM J. Imaging Sci. 2(1), 323–343 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  18. 18.
    Lai, M., Xu, Y., Yin, W.: Improved iteratively reweighted least squares for unconstrained smoothed \(\ell _q\) minimization. SIAM J. Numer. Anal. 51(2), 927–957 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  19. 19.
    Le Thi, H.A., Thi, B.T.N., Le, H.M.: Sparse signal recovery by difference of convex functions algorithms. In: Selamat, A., Nguyen, N.T., Haron, H. (eds.) Intelligent Information and Database Systems, pp. 387–397. Springer, Berlin (2013)Google Scholar
  20. 20.
    Le Thi, H.A., Huynh, V.N., Dinh, T.: DC programming and DCA for general DC programs. In: Do, T.V., Le Thi, H.A. Nguyen, N.T. (eds.) Advanced Computational Methods for Knowledge Engineering, pp. 15–35. Springer, Berlin (2014)Google Scholar
  21. 21.
    Le Thi, H.A., Pham Dinh, T., Le, H.M., Vo, X.T.: DC approximation approaches for sparse optimization. Eur. J. Oper. Res. 244(1), 26–46 (2015)MathSciNetCrossRefzbMATHGoogle Scholar
  22. 22.
    Lou, Y., Yin, P., Xin, J.: Point source super-resolution via non-convex L1 based methods. J. Sci. Comput. 68(3), 1082–1100 (2016)MathSciNetCrossRefzbMATHGoogle Scholar
  23. 23.
    Lou, Y., Yin, P., He, Q., Xin, J.: Computing sparse representation in a highly coherent dictionary based on difference of L1 and L2. J. Sci. Comput. 64, 178–196 (2015)MathSciNetCrossRefzbMATHGoogle Scholar
  24. 24.
    Lu, Z., Zhang, Y.: Sparse approximation via penalty decomposition methods. SIAM J. Optim. 23(4), 2448–2478 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  25. 25.
    Lv, J., Fan, Y.: A unified approach to model selection and sparse recovery using regularized least squares. Ann. Stat. 37(6A), 3498–3528 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  26. 26.
    Mallat, S., Zhang, Z.: Matching pursuits with time-frequency dictionaries. IEEE Trans. Signal Process. 41(12), 3397–3415 (1993)CrossRefzbMATHGoogle Scholar
  27. 27.
    Mazumder, R., Friedman, J., Hastie, T.: SparseNet: coordinate descent with nonconvex penalties. J. Am. Stat. Assoc. 106(495), 1125–1138 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  28. 28.
    Natarajan, B.: Sparse approximate solutions to linear systems. SIAM J. Comput. 24(2), 227–234 (1995)MathSciNetCrossRefzbMATHGoogle Scholar
  29. 29.
    Needell, D., Vershynin, R.: Signal recovery from incomplete and inaccurate measurements via regularized orthogonal matching pursuit. IEEE J. Sel. Top. Signal Process. 4(2), 310–316 (2010)CrossRefGoogle Scholar
  30. 30.
    Nguyen, T.B.T., Le Thi, H.A., Le, H.M., Vo, X.T.: DC approximation approach for \(\ell _0\)-minimization in compressed sensing. In: Do, T.V., Le Thi, H.A. , Nguyen N.T. ( eds.) Advanced Computational Methods for Knowledge Engineering, pp. 37–48. Springer, Berlin (2015)Google Scholar
  31. 31.
    Nikolova, M.: Local strong homogeneity of a regularized estimator. SIAM J. Appl. Math. 61(2), 633–658 (2000)MathSciNetCrossRefzbMATHGoogle Scholar
  32. 32.
    Ong, C.S., Le Thi, H.A.: Learning sparse classifiers with difference of convex functions algorithms. Optim. Methods Softw. 28(4), 830–854 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  33. 33.
    Pham Dinh, T., Le Thi, A.: Convex analysis approach to d.c. programming: theory, algorithms and applications. Acta Math. Vietnam. 22(1), 289–355 (1997)MathSciNetzbMATHGoogle Scholar
  34. 34.
    Pham Dinh, T., Le Thi, H.A.: A DC optimization algorithm for solving the trust-region subproblem. SIAM J. Optim. 8(2), 476–505 (1998)MathSciNetCrossRefzbMATHGoogle Scholar
  35. 35.
    Soubies, E., Blanc-Féraud, L., Aubert, G.: A continuous exact \(\ell _0\) penalty (CEL0) for least squares regularized problem. SIAM J. Imaging Sci. 8(3), 1607–1639 (2015)MathSciNetCrossRefzbMATHGoogle Scholar
  36. 36.
    Tibshirani, R.: Regression shrinkage and selection via the lasso. J. R. Stat. Soc. 58(1), 267–288 (1996)MathSciNetzbMATHGoogle Scholar
  37. 37.
    Tran, H., Webster, C.: Unified sufficient conditions for uniform recovery of sparse signals via nonconvex minimizations. arXiv:1701.07348. 19 Oct 2017
  38. 38.
    Tropp, J., Gilbert, A.: Signal recovery from partial information via orthogonal matching pursuit. IEEE Trans. Inf. Theory 53(12), 4655–4666 (2007)CrossRefzbMATHGoogle Scholar
  39. 39.
    Xu, F., Wang, S.: A hybrid simulated annealing thresholding algorithm for compressed sensing. Signal Process. 93, 1577–1585 (2013)CrossRefGoogle Scholar
  40. 40.
    Xu, Z., Chang, X., Xu, F., Zhang, H.: \(L_{1/2}\) regularization: a thresholding representation theory and a fast solver. IEEE Trans. Neural Netw. Learn. Syst. 23(7), 1013–1027 (2012)CrossRefGoogle Scholar
  41. 41.
    Yang, J., Zhang, Y.: Alternating direction algorithms for \(l_1\) problems in compressive sensing. SIAM J. Sci. Comput. 33(1), 250–278 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  42. 42.
    Yin, P., Lou, Y., He, Q., Xin, J.: Minimization of \(\ell _{1-2}\) for compressed sensing. SIAM J. Sci. Comput. 37(1), A536–A563 (2015)MathSciNetCrossRefzbMATHGoogle Scholar
  43. 43.
    Yin, P., Xin, J.: Iterative \(\ell _1\) minimization for non-convex compressed sensing. J. Comput. Math. 35(4), 437–449 (2017)CrossRefGoogle Scholar
  44. 44.
    Yin, W., Osher, S., Goldfarb, D., Darbon, J.: Bregman iterative algorithms for \(l_1\)-minimization with applications to compressed sensing. SIAM J. Imaging Sci. 1(1), 143–168 (2008)MathSciNetCrossRefzbMATHGoogle Scholar
  45. 45.
    Zeng, J., Lin, S., Wang, Y., Xu, Z.: \(L_{1/2}\) regularization: convergence of iterative half thresholding algorithm. IEEE Trans. Signal Process. 62(9), 2317–2329 (2014)MathSciNetCrossRefGoogle Scholar
  46. 46.
    Zhang, C.: Nearly unbiased variable selection under minimax concave penalty. Ann. Stat. 38, 894–942 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  47. 47.
    Zhang, S., Xin, J.: Minimization of transformed \(L_1\) penalty: closed form representation and iterative thresholding algorithms. Commun. Math. Sci. 15(2), 511–537 (2017)CrossRefzbMATHGoogle Scholar
  48. 48.
    Zhang, S., Yin, P., Xin, J.: Transformed Schatten-1 Iterative thresholding algorithms for low rank matrix completion. Commun. Math. Sci. 15(3), 839–862 (2017)MathSciNetCrossRefzbMATHGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature and Mathematical Optimization Society 2018

Authors and Affiliations

  1. 1.Department of MathematicsUniversity of CaliforniaIrvineUSA

Personalised recommendations