Dictionary Learning Problem

  • Bogdan Dumitrescu
  • Paul Irofti
Chapter
First Online:

Abstract

Dictionary learning can be formulated as an optimization problem in several ways. We present here the basic form, where the representation error is minimized under the constraint of sparsity, and discuss several views and relations with other data analysis and signal processing problems. We study some properties of the DL problem and their implications for the optimization process and explain the two subproblems that are crucial in DL algorithms: sparse coding and dictionary update. In preparation to algorithms analysis and comparisons, we present the main test problems, dealing with representation error and dictionary recovery; we give all details of test procedures, using either artificial data or images. Finally, as an appetizer for the remainder of the book, we illustrate the wide use of DL algorithms in the context of sparse representations for several applications like denoising, inpainting, compression, compressed sensing, classification.

This is a preview of subscription content, log in to check access

References

  1. 3.
    A. Agarwal, A. Anandkumar, P. Jain, P. Netrapalli, R. Tandon, Learning sparsely used overcomplete dictionaries, in Conference on Learning Theory (2014), pp. 123–137Google Scholar
  2. 4.
    M. Aharon, M. Elad, A. Bruckstein, K-SVD: an algorithm for designing overcomplete dictionaries for sparse representation. IEEE Trans. Signal Process. 54(11), 4311–4322 (2006)Google Scholar
  3. 5.
    M. Aharon, M. Elad, A. Bruckstein, On the uniqueness of overcomplete dictionaries, a practical way to retrieve them. Linear Algebra Appl. 416(1), 48–67 (2006)Google Scholar
  4. 6.
    S. Arora, R. Ge, A. Moitra, New algorithms for learning incoherent overcomplete dictionaries, in Conference on Learning Theory (2014), pp. 779–806Google Scholar
  5. 18.
    O. Bryt, M. Elad, Compression of facial images using the K-SVD algorithm. J. Vis. Commun. Image Represent. 19(4), 270–282 (2008)Google Scholar
  6. 24.
    E.J. Candès, L. Wakin, An introduction to compressive sampling. IEEE Signal Process. Mag. 25(2), 21–30 (2008)Google Scholar
  7. 25.
    E.J. Candès, J. Romberg, T. Tao, Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information. IEEE Trans. Inf. Theory 52(2), 489–509 (2006)Google Scholar
  8. 32.
    H. Chang, M.K. Ng, T. Zeng, Reducing artifacts in JPEG decompression via a learned dictionary. IEEE Trans Signal Process. 62(3), 718–728 (2014)Google Scholar
  9. 39.
    X. Chen, Z. Du, J. Li, X. Li, H. Zhang, Compressed sensing based on dictionary learning for extracting impulse components. Signal Process. 96, 94–109 (2014)Google Scholar
  10. 51.
    D.L. Donoho, Compressed sensing. IEEE Trans. Inf. Theory 52(4), 1289–1306 (2006)Google Scholar
  11. 54.
    D.L. Donoho, J.M. Johnstone, Ideal spatial adaptation by wavelet shrinkage. Biometrika 81(3), 425–455 (1994)Google Scholar
  12. 56.
    M.F. Duarte, M.A. Davenport, D. Takbar, J.N. Laska, T. Sun, K.F. Kelly, R.G. Baraniuk, Single-pixel imaging via compressive sampling. IEEE Signal Process. Mag. 25(2), 83–91 (2008)Google Scholar
  13. 57.
    J.M. Duarte-Carvajalino, G. Sapiro, Learning to sense sparse signals: simultaneous sensing matrix and sparsifying dictionary optimization. IEEE Trans. Image Process. 18(7), 1395–1408 (2009)Google Scholar
  14. 62.
    M. Elad, M. Aharon, Image denoising via sparse and redundant representations over learned dictionaries. IEEE Trans. Image Process. 15(12), 3736–3745 (2006)Google Scholar
  15. 65.
    M. Elad, J.L. Starck, P. Querre, D.L. Donoho, Simultaneous cartoon and texture image inpainting using morphological component analysis (MCA). Appl. Comput. Harmon. Anal. 19(3), 340–358 (2005)Google Scholar
  16. 69.
    M.J. Fadili, J.L. Starck, F. Murtagh, Inpainting and zooming using sparse representations. Comput. J. 52(1), 64–79 (2007)Google Scholar
  17. 77.
    A. Gersho, R.M. Gray, Vector Quantization and Signal Compression (Springer Science & Business Media, Berlin, 1992)Google Scholar
  18. 88.
    C. Guillemot, O. Le Meur, Image Inpainting. IEEE Signal Process. Mag. 31(1), 127–144 (2014)Google Scholar
  19. 89.
    O.G. Guleryuz, Nonlinear approximation based image recovery using adaptive sparse reconstructions and iterated denoising-Part II: adaptive algorithms. IEEE Trans. Image Process. 15(3), 555–571 (2006)Google Scholar
  20. 106.
    C. Jung, L. Jiao, H. Qi, T. Sun, Image deblocking via sparse representation. Signal Process. Image Commun. 27(6), 663–677 (2012)Google Scholar
  21. 122.
    S. Lloyd, Least squares quantization in PCM. IEEE Trans. Info. Theory 28(2), 129–137 (1982)Google Scholar
  22. 127.
    J. Mairal, M. Elad, G. Sapiro, Sparse representation for color image restoration. IEEE Trans. Image Process. 17(1), 53–69 (2008)Google Scholar
  23. 132.
    K. Marwah, G. Wetzstein, Y. Bando, R. Raskar, Compressive light field photography using overcomplete dictionaries and optimized projections. ACM Trans. Graph. 32(4), 46 (2013)Google Scholar
  24. 136.
    J.F. Murray, K. Kreutz-Delgado, Sparse image coding using learned overcomplete dictionaries, in International Workshop on Machine Learning for Signal Processing (2004), pp. 579–588Google Scholar
  25. 141.
    B.A. Olshausen, D.J. Field, Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature 381, 607–609 (1996)Google Scholar
  26. 147.
    J. Portilla, V. Strela, M.J. Wainwright, E.P. Simoncelli, Image denoising using scale mixtures of Gaussians in the wavelet domain. IEEE Trans. Image Process. 12(11), 1338–1351 (2003)Google Scholar
  27. 153.
    S. Ravishankar, Y. Bresler, MR image reconstruction from highly undersampled k-space data by dictionary learning. IEEE Trans. Med. Imag. 30(5), 1028–1041 (2011)Google Scholar
  28. 158.
    R. Rubinstein, A.M. Bruckstein, M. Elad, Dictionaries for sparse representations modeling. Proc. IEEE 98(6), 1045–1057 (2010)Google Scholar
  29. 172.
    L. Shao, R. Yan, X. Li, Y. Liu, From heuristic optimization to dictionary learning: a review and comprehensive comparison of image denoising algorithms. IEEE Trans. Cybern. 44(7), 1001–1013 (2014)Google Scholar
  30. 178.
    K. Skretting, K. Engan, Image compression using learned dictionaries by RLS-DLA and compared with K-SVD, in International Conference on Acoustics Speech Signal Processing (ICASSP), Prague (2011), pp. 1517–1520Google Scholar
  31. 181.
    J.L. Starck, E.J. Candès, D.L. Donoho, The curvelet transform for image denoising. IEEE Trans. Image Process. 11(6), 670–684 (2002)Google Scholar
  32. 184.
    J. Sun, Q. Qu, J. Wright, Complete dictionary recovery over the sphere, in International Conference on Sampling Theory and Applications (SampTA) (2015), pp. 407–410Google Scholar
  33. 189.
    I. Tosic, P. Frossard, Dictionary learning. IEEE Signal Process. Mag. 28(2), 27–38 (2011)Google Scholar
  34. 193.
    D. Vainsencher, S. Mannor, A.M. Bruckstein, The sample complexity of dictionary learning. J. Mach. Learn. Res. 12, 3259–3281 (2011)Google Scholar
  35. 198.
    Z. Wang, A.C. Bovik, H.R. Sheikh, E.P. Simoncelli, Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)Google Scholar
  36. 202.
    A.G. Weber, The USC-SIPI image database (1997)Google Scholar
  37. 209.
    M. Xu, X. Jia, M. Pickering, A.J. Plaza, Cloud removal based on sparse representation via multitemporal dictionary learning. IEEE Trans. Geosci. Remote Sens. 54(5), 2998–3006 (2016)Google Scholar
  38. 221.
    Q. Zhang, B. Li, Dictionary Learning in Visual Computing (Morgan & Claypool, San Rafael, 2015)Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Bogdan Dumitrescu
    • 1
  • Paul Irofti
    • 2
  1. 1.Department of Automatic Control and Systems Engineering, Faculty of Automatic Control and ComputersUniversity Politehnica of BucharestBucharestRomania
  2. 2.Department of Computer Science, Faculty of Mathematics and Computer ScienceUniversity of BucharestBucharestRomania

Personalised recommendations