Abstract
The sparse coding method is formulated as an information theoretic optimization problem. The rate distortion theory leads to an objective functional which can be interpreted as an information theoretic formulation of the sparse coding. Viewing as an entropy minimization problem, the rate distortion theory and consequently the sparse coding are extended to discriminative variants. As a concrete example of this information theoretic sparse coding, a discriminative non-linear sparse coding algorithm with neural networks is proposed. Experimental results of gender classification by face images show that the discriminative sparse coding is more robust to noise, compared to the conventional method which directly uses images as inputs to a linear support vector machine.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Salomon, D.: Data Compression: The Complete Reference. Springer, Heidelberg (2004)
Olshausen, B.A., Field, D.J.: Sparse Coding of Sensory Inputs. Current Opinion in Neurobiology 14(4), 481–487 (2004)
Cover, T.M., Thomas, J.A.: Elements of Information Theory. John Wiley and Sons, Inc., Chichester (1991)
Aharon, M., Elad, M., Bruckstein, A.: K-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation. IEEE Trans. Signal Processing 54(11), 4311–4322 (2006)
Tishby, N., Pereira, F.C., Bialek, W.: The Information Bottleneck Method. In: 37-th Annual Allerton Conference on Communication, Control and Computing, pp. 368–377 (1999)
Elad, M., Aharon, M.: Image Denoising via Sparse and Redundant Representations over Learned Dictionaries. IEEE Trans. Image Processing 15(12), 3736–3745 (2006)
Phillips, P.J., Moon, H., Rizvi, S.A., Rauss, P.J.: The FERET Evaluation Methodology for Face-recognition Algorithms. IEEE Trans. Pattern Anal. Mach. Intell. 22(10), 1090–1104 (2000)
Yang, M.H., Moghaddam, B.: Support Vector Machines for Visual Gender Classification. In: International Conference on Pattern Recognition, vol. 1, p. 5115 (2000)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2009 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Hino, H., Murata, N. (2009). An Information Theoretic Perspective of the Sparse Coding. In: Yu, W., He, H., Zhang, N. (eds) Advances in Neural Networks – ISNN 2009. ISNN 2009. Lecture Notes in Computer Science, vol 5551. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-01507-6_11
Download citation
DOI: https://doi.org/10.1007/978-3-642-01507-6_11
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-01506-9
Online ISBN: 978-3-642-01507-6
eBook Packages: Computer ScienceComputer Science (R0)