Abstract
Sparse coding employs low-dimensional subspaces in order to encode high-dimensional signals. Finding the optimal subspaces is a difficult optimization task. We show that stochastic gradient descent is superior in finding the optimal subspaces compared to MOD and K-SVD, which are both state-of-the art methods. The improvement is most significant in the difficult setting of highly overlapping subspaces. We introduce the so-called “Bag of Pursuits” that is derived from Orthogonal Matching Pursuit. It provides an improved approximation of the optimal sparse coefficients, which, in turn, significantly improves the performance of the gradient descent approach as well as MOD and K-SVD. In addition, the “Bag of Pursuits” allows to employ a generalized version of the Neural Gas algorithm for sparse coding, which finally leads to an even more powerful method.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Referencs
AHARON, M., ELAD, M. and BRUCKSTEIN. A. (2006): K-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation. IEEE Transactions on Signal Processing,[see also IEEE Transactions on Acoustics, Speech, and Signal Processing], 54(11):4311–4322.
DAVIS, G., MALLAT, S. and AVELLANEDA. M. (1997): Greedy adaptive approximation. J. Constr. Approx., 13:57–89.
K. ENGAN, K., AASE, S. O. and HAKON HUSOY., J. (1999): Method of optimal directions for frame design. In ICASSP ’99: Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing, pages 2443–2446, Washington, DC, USA, 1999. IEEE Computer Society.
LABUSCH, K., BARTH, E. and MARTINETZ. T. (2009): Sparse Coding Neural Gas: Learning of Overcomplete Data Representations. Neurocomputing, 72(7-9):1547–1555.
MARTINETZ, T., BERKOVICH, S. and SCHULTEN. K. (1993): “Neural-gas” Network for Vector Quantization and its Application to Time-Series Prediction. IEEE-Transactions on Neural Networks, 4(4):558–569.
PATI, Y., REZAIIFAR, R. and KRISHNAPRASAD. P. (1993): Orthogonal Matching Pursuit: Recursive Function Approximation with Applications to Wavelet Decomposition. Proceedings of the 27 th Annual Asilomar Conference on Signals, Systems, and Computers.
REBOLLO-NEIRA, L. and LOWE. D. (2002): Optimized orthogonal matching pursuit approach. IEEE Signal Processing Letters, 9(4):137–140.
TROPP. J. A. (2004): Greed is good: algorithmic results for sparse approximation. IEEE Transactions on Information Theory, 50(10):2231–2242.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2010 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Labusch, K., Barth, E., Martinetz, T. (2010). Bag of Pursuits and Neural Gas for Improved Sparse Coding. In: Lechevallier, Y., Saporta, G. (eds) Proceedings of COMPSTAT'2010. Physica-Verlag HD. https://doi.org/10.1007/978-3-7908-2604-3_30
Download citation
DOI: https://doi.org/10.1007/978-3-7908-2604-3_30
Published:
Publisher Name: Physica-Verlag HD
Print ISBN: 978-3-7908-2603-6
Online ISBN: 978-3-7908-2604-3
eBook Packages: Mathematics and StatisticsMathematics and Statistics (R0)