Advertisement

Sparse Representations Are Most Likely to Be the Sparsest Possible

  • Michael Elad
Open Access
Research Article
Part of the following topical collections:
  1. Frames and Overcomplete Representations in Signal Processing, Communications, and Information Theory

Abstract

Given a signal Open image in new window and a full-rank matrix Open image in new window with Open image in new window , we define the signal's overcomplete representations as all Open image in new window satisfying Open image in new window . Among all the possible solutions, we have special interest in the sparsest one—the one minimizing Open image in new window . Previous work has established that a representation is unique if it is sparse enough, requiring Open image in new window . The measure Open image in new window stands for the minimal number of columns from Open image in new window that are linearly dependent. This bound is tight—examples can be constructed to show that with Open image in new window or more nonzero entries, uniqueness is violated. In this paper we study the behavior of overcomplete representations beyond the above bound. While tight from a worst-case standpoint, a probabilistic point of view leads to uniqueness of representations satisfying Open image in new window . Furthermore, we show that even beyond this point, uniqueness can still be claimed with high confidence. This new result is important for the study of the average performance of pursuit algorithms—when trying to show an equivalence between the pursuit result and the ideal solution, one must also guarantee that the ideal result is indeed the sparsest.

Keywords

Information Technology Special Interest Average Performance Quantum Information High Confidence 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

References

  1. 1.
    Chen SS, Donoho DL, Saunders MA: Atomic decomposition by basis pursuit. SIAM Review 2001, 43(1):129-159. 10.1137/S003614450037906XMathSciNetCrossRefMATHGoogle Scholar
  2. 2.
    Natarajan BK: Sparse approximate solutions to linear systems. SIAM Journal on Computing 1995, 24(2):227-234. 10.1137/S0097539792240406MathSciNetCrossRefMATHGoogle Scholar
  3. 3.
    Davis G, Mallat SG, Avellaneda M: Adaptive greedy approximations. Constructive Approximations 1997, 13(1):57-98.MathSciNetCrossRefMATHGoogle Scholar
  4. 4.
    Mallat SG, Zhang Z: Matching pursuits with time-frequency dictionaries. IEEE Transactions on Signal Processing 1993, 41(12):3397-3415. 10.1109/78.258082CrossRefMATHGoogle Scholar
  5. 5.
    Pati YC, Rezaiifar R, Krishnaprasad PS: Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition. Proceedings of 27th Asilomar Conference on Signals, Systems and Computers, November 1993, Pacific Grove, Calif, USA 1: 40-44.CrossRefGoogle Scholar
  6. 6.
    Couvreur C, Bresler Y: On the optimality of the backward greedy algorithm for the subset selection problem. SIAM Journal on Matrix Analysis and Applications 2000, 21(3):797-808. 10.1137/S0895479898332928MathSciNetCrossRefMATHGoogle Scholar
  7. 7.
    Gilbert AC, Muthukrishnan S, Strauss MJ: Approximation of functions over redundant dictionaries using coherence. Proceedings of 14th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA '03), January 2003, Baltimore, Md, USA 243-252.Google Scholar
  8. 8.
    Tropp JA, Gilbert AC, Muthukrishnan S, Strauss MJ: Improved sparse approximation over quasi-incoherent dictionaries. Proceedings of IEEE International Conference on Image Processing (ICIP '03), September 2003, Barcelona, Spain 1: 37-40.Google Scholar
  9. 9.
    Tropp JA: Greed is good: algorithmic results for sparse approximation. IEEE Transactions on Information Theory 2004, 50(10):2231-2242. 10.1109/TIT.2004.834793MathSciNetCrossRefMATHGoogle Scholar
  10. 10.
    Donoho DL, Elad M, Temlyakov VN: Stable recovery of sparse overcomplete representations in the presence of noise. to appear in IEEE Transactions on Information Theory, February 2004MATHGoogle Scholar
  11. 11.
    Tropp JA: Just relax: Convex programming methods for subset selection and sparse approximation. In ICES Report 04-04. University of Texas at Austin, Austin, Tex, USA; February 2004.Google Scholar
  12. 12.
    Temlyakov VN:Greedy algorithms andOpen image in new window-term approximation with regard to redundant dictionaries. Journal of Approximation Theory 1999, 98(1):117-145. 10.1006/jath.1998.3265MathSciNetCrossRefMATHGoogle Scholar
  13. 13.
    Temlyakov VN: Weak greedy algorithms. Advances in Computational Mathematics 2000, 12(2-3):213-227.MathSciNetCrossRefMATHGoogle Scholar
  14. 14.
    Gorodnitsky IF, Rao BD: Sparse signal reconstruction from limited data using FOCUSS: A re-weighted minimum norm algorithm. IEEE Transactions on Signal Processing 1997, 45(3):600-616. 10.1109/78.558475CrossRefGoogle Scholar
  15. 15.
    Donoho DL, Huo X: Uncertainty principles and ideal atomic decomposition. IEEE Transactions on Information Theory 2001, 47(7):2845-2862. 10.1109/18.959265MathSciNetCrossRefMATHGoogle Scholar
  16. 16.
    Huo X: Sparse image representation via combined transforms, M.S. thesis. Stanford University, Stanford, Calif, USA; 1999.Google Scholar
  17. 17.
    Elad M, Bruckstein AM: A generalized uncertainty principle and sparse representation in pairs of bases. IEEE Transactions on Information Theory 2002, 48(9):2558-2567. 10.1109/TIT.2002.801410MathSciNetCrossRefMATHGoogle Scholar
  18. 18.
    Donoho DL, Elad M:Optimally sparse representation in general (non-orthogonal) dictionaries viaOpen image in new window minimization. Proceedings of the National Academy of Sciences of the United States of America 2003, 100(5):2197-2202. 10.1073/pnas.0437847100MathSciNetCrossRefMATHGoogle Scholar
  19. 19.
    Gribonval R, Nielsen M: Sparse representations in unions of bases. IEEE Transactions on Information Theory 2003, 49(12):3320-3325. 10.1109/TIT.2003.820031MathSciNetCrossRefMATHGoogle Scholar
  20. 20.
    Fuchs J-J: On sparse representations in arbitrary redundant bases. IEEE Transactions on Information Theory 2004, 50(6):1341-1344. 10.1109/TIT.2004.828141CrossRefMathSciNetMATHGoogle Scholar
  21. 21.
    Candès EJ, Romberg J: Quantitative robust uncertainty principles and optimally sparse decompositions. to appear in Foundations of Computational Mathematics, 2004MATHGoogle Scholar
  22. 22.
    Donoho DL:For most large underdetermined systems of linear equations the minimalOpen image in new window-norm solution is also the sparsest solution. In Tech. Rep. 2004-10. Department of Statistics, Stanford University, Stanford, Calif, USA; September 2004.Google Scholar
  23. 23.
    Edelman A: Eigenvalues and condition numbers of random matrices. SIAM Journal on Matrix Analysis and Applications 1988, 9(4):543-560. 10.1137/0609045MathSciNetCrossRefMATHGoogle Scholar
  24. 24.
    Shen J: On the singular values of Gaussian random matrices. Linear Algebra and its Applications 2001, 326(1–3):1-14.MathSciNetCrossRefMATHGoogle Scholar
  25. 25.
    Strohmer T, Heath RW Jr.: Grassmannian frames with applications to coding and communication. Applied and Computational Harmonic Analysis 2003, 14(3):257-275. 10.1016/S1063-5203(03)00023-XMathSciNetCrossRefMATHGoogle Scholar
  26. 26.
    Tropp JA, Dhillon IS, Heath RW Jr., Strohmer T: Designing structured tight frames via an alternating projection method. IEEE Transactions on Information Theory 2005, 51(1):188-209.MathSciNetCrossRefMATHGoogle Scholar
  27. 27.
    Candès EJ, Donoho DL:New tight frames of curvelets and the problem of approximating piecewiseOpen image in new window images with piecewiseOpen image in new window edges. 2002.Google Scholar
  28. 28.
    Starck J-L, Candès EJ, Donoho DL: The curvelet transform for image denoising. IEEE Transactions on Image Processing 2002, 11(6):670-684. 10.1109/TIP.2002.1014998MathSciNetCrossRefMATHGoogle Scholar
  29. 29.
    Björner A: Some matroid inequalities. Discrete Mathematics 1980, 31(1):101-103. 10.1016/0012-365X(80)90179-XMathSciNetCrossRefMATHGoogle Scholar
  30. 30.
    Goldberg F: Overcomplete Bases, Spark, and Matroids. 2004.Google Scholar

Copyright information

© Elad. 2006

Authors and Affiliations

  • Michael Elad
    • 1
  1. 1.Computer Science DepartmentThe Technion – Israel Institute of TechnologyHaifaIsrael

Personalised recommendations