Regularization and Incoherence

  • Bogdan Dumitrescu
  • Paul Irofti


A dictionary should be faithful to the signals it represents, in the sense that the sparse representation error in learning is small, but also must be reliable when recovering sparse representations. A direct way to obtain good recovery guarantees is to modify the objective of the DL optimization such that the resulted dictionary is incoherent, meaning that the atoms are generally far from each other. Alternatively, we can explicitly impose mutual coherence bounds. A related modification of the objective is regularization, in the usual form encountered in least squares problems. Regularization has the additional benefit of making the DL process avoid bottlenecks generated by ill-conditioned dictionaries. We present several algorithms for regularization and promoting incoherence and illustrate their benefits. The K-SVD family can be adapted to such approaches, with good results in the case of regularization. Other methods for obtaining incoherence are based on the gradient of a combined objective including the frame potential or by including a decorrelation step in the standard algorithms. Such a step is based on successive projections on two sets whose properties are shared by the Gram matrix corresponding to the dictionary, which reduce mutual coherence, and rotations of the dictionary, which regain the adequacy to the training signals. We also give a glimpse on the currently most efficient methods that aim at the minimization of the mutual coherence of a frame, regardless of the training signals.


  1. 10.
    D. Barchiesi, M.D. Plumbley, Learning incoherent dictionaries for sparse approximation using iterative projections and rotations. IEEE Trans. Signal Process. 61(8), 2055–2065 (2013)Google Scholar
  2. 27.
    P.G. Casazza, R.G. Lynch, A brief introduction to Hilbert space frame theory and its applications, in Proceedings of Symposia in Applied Mathematics, ed. by K.A. Okoudjou, vol. 73 (American Mathematical Society, Providence, 2016), pp. 1–52Google Scholar
  3. 38.
    W. Chen, M.R.D. Rodrigues, I.J. Wassell, On the use of unit-norm tight frames to improve the average MSE performance in compressive sensing applications. IEEE Signal Process. Lett. 19(1), 8–11 (2012)CrossRefGoogle Scholar
  4. 44.
    W. Dai, T. Xu, W. Wang, Simultaneous codeword optimization (SimCO) for dictionary update and learning. IEEE Trans. Signal Process. 60(12), 6340–6353 (2012)Google Scholar
  5. 49.
    P. Davies, N.J. Higham, Numerically stable generation of correlation matrices and their factors. BIT Numer. Math. 40(4), 640–651 (2000)Google Scholar
  6. 58.
    B. Dumitrescu, Designing incoherent frames with only matrix-vector multiplications. IEEE Signal Process. Lett. 24(9), 1265–1269 (2017)CrossRefGoogle Scholar
  7. 60.
    B. Dumitrescu, P. Irofti, Regularized K-SVD. IEEE Signal Process. Lett. 24(3), 309–313 (2017)CrossRefGoogle Scholar
  8. 100.
    P. Irofti, B. Dumitrescu, Regularized algorithms for dictionary learning, in Proceedings of International Conference Communications, Bucharest (2016), pp. 439–442Google Scholar
  9. 120.
    T. Lin, S. Liu, H. Zha, Incoherent dictionary learning for sparse representation, in 21st International Conference on Pattern Recognition (ICPR), Tsukuba, November 2012, pp. 1237–1240Google Scholar
  10. 133.
    J.L. Massey, T. Mittelholzer, Welch’s bound and sequence sets for code-division multiple-access systems, in Sequences II, ed. by R. Capocelli, A. De Santis, U. Vaccaro (Springer, Berlin, 1993), pp. 63–78CrossRefGoogle Scholar
  11. 138.
    D. Needell R. Vershynin, Uniform uncertainty principle and signal recovery via regularized orthogonal matching pursuit. Found. Comput. Math. 9(3), 317–334 (2009)MathSciNetCrossRefGoogle Scholar
  12. 139.
    M. Nejati, S. Samavi, N. Karimi, S.M.R. Soroushmehr, K. Najarian, Boosted dictionary learning for image compression. IEEE Trans. Image Process. 25(10), 4900–4915 (2016)MathSciNetCrossRefGoogle Scholar
  13. 163.
    C. Rusu, N. Gonzalez-Prelcic, Designing incoherent frames through convex techniques for optimized compressed sensing. IEEE Trans. Signal Process. 64(9), 2334–2344 (2016)MathSciNetCrossRefGoogle Scholar
  14. 166.
    M. Sadeghi, M. Babaie-Zadeh, Incoherent unit-norm frame design via an alternating minimization penalty method. IEEE Signal Process. Lett. 24(1), 32–36 (2017)CrossRefGoogle Scholar
  15. 176.
    C.D. Sigg, T. Dikk, J.D. Buhmann, Learning dictionaries with bounded self-coherence. IEEE Signal Process. Lett. 19(19), 861–865 (2012)CrossRefGoogle Scholar
  16. 191.
    J.A. Tropp, I.S. Dhillon, R.W. Heath Jr., T. Strohmer, Designing structured tight frames via an alternating projection method. IEEE Trans. Inf. Theory 51(1), 188–209 (2005)MathSciNetCrossRefGoogle Scholar
  17. 192.
    E.V. Tsiligianni, L.P. Kondi, A.K. Katsaggelos, Construction of incoherent unit norm tight frames with application to compressed sensing. IEEE Trans. Info. Theory 60(4), 2319–2330 (2014)MathSciNetCrossRefGoogle Scholar
  18. 197.
    S. Waldron, Generalized Welch bound equality sequences are tight frames. IEEE Trans. Inf. Theory 49(9), 2307–2309 (2003)MathSciNetCrossRefGoogle Scholar
  19. 203.
    L. Welch, Lower bounds on the maximum cross correlation of signals. IEEE Trans. Inf. Theory 20(3), 397–399 (1974)CrossRefGoogle Scholar
  20. 211.
    M. Yaghoobi, L. Daudet, M.E. Davies, Parametric dictionary design for sparse coding. IEEE Trans. Signal Process. 57(12), 4800–4810 (2009)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Bogdan Dumitrescu
    • 1
  • Paul Irofti
    • 2
  1. 1.Department of Automatic Control and Systems Engineering, Faculty of Automatic Control and ComputersUniversity Politehnica of BucharestBucharestRomania
  2. 2.Department of Computer Science, Faculty of Mathematics and Computer ScienceUniversity of BucharestBucharestRomania

Personalised recommendations