Dictionary Learning Problem
Abstract
Dictionary learning can be formulated as an optimization problem in several ways. We present here the basic form, where the representation error is minimized under the constraint of sparsity, and discuss several views and relations with other data analysis and signal processing problems. We study some properties of the DL problem and their implications for the optimization process and explain the two subproblems that are crucial in DL algorithms: sparse coding and dictionary update. In preparation to algorithms analysis and comparisons, we present the main test problems, dealing with representation error and dictionary recovery; we give all details of test procedures, using either artificial data or images. Finally, as an appetizer for the remainder of the book, we illustrate the wide use of DL algorithms in the context of sparse representations for several applications like denoising, inpainting, compression, compressed sensing, classification.
References
- 3.A. Agarwal, A. Anandkumar, P. Jain, P. Netrapalli, R. Tandon, Learning sparsely used overcomplete dictionaries, in Conference on Learning Theory (2014), pp. 123–137Google Scholar
- 4.M. Aharon, M. Elad, A. Bruckstein, K-SVD: an algorithm for designing overcomplete dictionaries for sparse representation. IEEE Trans. Signal Process. 54(11), 4311–4322 (2006)Google Scholar
- 5.M. Aharon, M. Elad, A. Bruckstein, On the uniqueness of overcomplete dictionaries, a practical way to retrieve them. Linear Algebra Appl. 416(1), 48–67 (2006)Google Scholar
- 6.S. Arora, R. Ge, A. Moitra, New algorithms for learning incoherent overcomplete dictionaries, in Conference on Learning Theory (2014), pp. 779–806Google Scholar
- 18.O. Bryt, M. Elad, Compression of facial images using the K-SVD algorithm. J. Vis. Commun. Image Represent. 19(4), 270–282 (2008)Google Scholar
- 24.E.J. Candès, L. Wakin, An introduction to compressive sampling. IEEE Signal Process. Mag. 25(2), 21–30 (2008)Google Scholar
- 25.E.J. Candès, J. Romberg, T. Tao, Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information. IEEE Trans. Inf. Theory 52(2), 489–509 (2006)Google Scholar
- 32.H. Chang, M.K. Ng, T. Zeng, Reducing artifacts in JPEG decompression via a learned dictionary. IEEE Trans Signal Process. 62(3), 718–728 (2014)Google Scholar
- 39.X. Chen, Z. Du, J. Li, X. Li, H. Zhang, Compressed sensing based on dictionary learning for extracting impulse components. Signal Process. 96, 94–109 (2014)Google Scholar
- 51.D.L. Donoho, Compressed sensing. IEEE Trans. Inf. Theory 52(4), 1289–1306 (2006)Google Scholar
- 54.D.L. Donoho, J.M. Johnstone, Ideal spatial adaptation by wavelet shrinkage. Biometrika 81(3), 425–455 (1994)Google Scholar
- 56.M.F. Duarte, M.A. Davenport, D. Takbar, J.N. Laska, T. Sun, K.F. Kelly, R.G. Baraniuk, Single-pixel imaging via compressive sampling. IEEE Signal Process. Mag. 25(2), 83–91 (2008)Google Scholar
- 57.J.M. Duarte-Carvajalino, G. Sapiro, Learning to sense sparse signals: simultaneous sensing matrix and sparsifying dictionary optimization. IEEE Trans. Image Process. 18(7), 1395–1408 (2009)Google Scholar
- 62.M. Elad, M. Aharon, Image denoising via sparse and redundant representations over learned dictionaries. IEEE Trans. Image Process. 15(12), 3736–3745 (2006)Google Scholar
- 65.M. Elad, J.L. Starck, P. Querre, D.L. Donoho, Simultaneous cartoon and texture image inpainting using morphological component analysis (MCA). Appl. Comput. Harmon. Anal. 19(3), 340–358 (2005)Google Scholar
- 69.M.J. Fadili, J.L. Starck, F. Murtagh, Inpainting and zooming using sparse representations. Comput. J. 52(1), 64–79 (2007)Google Scholar
- 77.A. Gersho, R.M. Gray, Vector Quantization and Signal Compression (Springer Science & Business Media, Berlin, 1992)Google Scholar
- 88.C. Guillemot, O. Le Meur, Image Inpainting. IEEE Signal Process. Mag. 31(1), 127–144 (2014)Google Scholar
- 89.O.G. Guleryuz, Nonlinear approximation based image recovery using adaptive sparse reconstructions and iterated denoising-Part II: adaptive algorithms. IEEE Trans. Image Process. 15(3), 555–571 (2006)Google Scholar
- 106.C. Jung, L. Jiao, H. Qi, T. Sun, Image deblocking via sparse representation. Signal Process. Image Commun. 27(6), 663–677 (2012)Google Scholar
- 122.S. Lloyd, Least squares quantization in PCM. IEEE Trans. Info. Theory 28(2), 129–137 (1982)Google Scholar
- 127.J. Mairal, M. Elad, G. Sapiro, Sparse representation for color image restoration. IEEE Trans. Image Process. 17(1), 53–69 (2008)Google Scholar
- 132.K. Marwah, G. Wetzstein, Y. Bando, R. Raskar, Compressive light field photography using overcomplete dictionaries and optimized projections. ACM Trans. Graph. 32(4), 46 (2013)Google Scholar
- 136.J.F. Murray, K. Kreutz-Delgado, Sparse image coding using learned overcomplete dictionaries, in International Workshop on Machine Learning for Signal Processing (2004), pp. 579–588Google Scholar
- 141.B.A. Olshausen, D.J. Field, Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature 381, 607–609 (1996)Google Scholar
- 147.J. Portilla, V. Strela, M.J. Wainwright, E.P. Simoncelli, Image denoising using scale mixtures of Gaussians in the wavelet domain. IEEE Trans. Image Process. 12(11), 1338–1351 (2003)Google Scholar
- 153.S. Ravishankar, Y. Bresler, MR image reconstruction from highly undersampled k-space data by dictionary learning. IEEE Trans. Med. Imag. 30(5), 1028–1041 (2011)Google Scholar
- 158.R. Rubinstein, A.M. Bruckstein, M. Elad, Dictionaries for sparse representations modeling. Proc. IEEE 98(6), 1045–1057 (2010)Google Scholar
- 172.L. Shao, R. Yan, X. Li, Y. Liu, From heuristic optimization to dictionary learning: a review and comprehensive comparison of image denoising algorithms. IEEE Trans. Cybern. 44(7), 1001–1013 (2014)Google Scholar
- 178.K. Skretting, K. Engan, Image compression using learned dictionaries by RLS-DLA and compared with K-SVD, in International Conference on Acoustics Speech Signal Processing (ICASSP), Prague (2011), pp. 1517–1520Google Scholar
- 181.J.L. Starck, E.J. Candès, D.L. Donoho, The curvelet transform for image denoising. IEEE Trans. Image Process. 11(6), 670–684 (2002)Google Scholar
- 184.J. Sun, Q. Qu, J. Wright, Complete dictionary recovery over the sphere, in International Conference on Sampling Theory and Applications (SampTA) (2015), pp. 407–410Google Scholar
- 189.I. Tosic, P. Frossard, Dictionary learning. IEEE Signal Process. Mag. 28(2), 27–38 (2011)Google Scholar
- 193.D. Vainsencher, S. Mannor, A.M. Bruckstein, The sample complexity of dictionary learning. J. Mach. Learn. Res. 12, 3259–3281 (2011)Google Scholar
- 198.Z. Wang, A.C. Bovik, H.R. Sheikh, E.P. Simoncelli, Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)Google Scholar
- 202.A.G. Weber, The USC-SIPI image database (1997)Google Scholar
- 209.M. Xu, X. Jia, M. Pickering, A.J. Plaza, Cloud removal based on sparse representation via multitemporal dictionary learning. IEEE Trans. Geosci. Remote Sens. 54(5), 2998–3006 (2016)Google Scholar
- 221.Q. Zhang, B. Li, Dictionary Learning in Visual Computing (Morgan & Claypool, San Rafael, 2015)Google Scholar