Abstract
The paper introduces a new framework for feature learning in classification motivated by information theory. We first systematically study the information structure and present a novel perspective revealing the two key factors in information utilization: class-relevance and redundancy. We derive a new information decomposition model where a novel concept called class-relevant redundancy is introduced. Subsequently a new algorithm called Conditional Informative Feature Extraction is formulated, which maximizes the joint class-relevant information by explicitly reducing the class-relevant redundancies among features. To address the computational difficulties in information-based optimization, we incorporate Parzen window estimation into the discrete approximation of the objective function and propose a Local Active Region method which substantially increases the optimization efficiency. To effectively utilize the extracted feature set, we propose a Bayesian MAP formulation for feature fusion, which unifies Laplacian Sparse Prior and Multivariate Logistic Regression to learn a fusion rule with good generalization capability. Realizing the inefficiency caused by separate treatment of the extraction stage and the fusion stage, we further develop an improved design of the framework to coordinate the two stages by introducing a feedback from the fusion stage to the extraction stage, which significantly enhances the learning efficiency. The results of the comparative experiments show remarkable improvements achieved by our framework.
Chapter PDF
Similar content being viewed by others
Keywords
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
References
Turk, M., Pentland, A.: Eigenfaces for Recognition. J. Cognitive Neuroscience 3(1), 71–86 (1991)
Belhumeur, P.N., Hespanha, J.P., Kriegman, D.J.: Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection. IEEE Trans. on PAMI 19(7), 711–720 (1997)
Etemad, K., Chellappa, R.: Discriminant Analysis for Recognition of Human Face Images. J. Opt. Soc. Am. 14(8), 1724–1733 (1997)
Wang, X., Tang, X.: A Unified Framework for Subspace Face Recognition. IEEE Trans. on PAMI 26(9), 1222–1228 (2004)
Duin, R.P.W., Haeb-Umbach, R.: Multiclass Linear Dimension Reduction by Weighted Pairwise Fisher Criteria. IEEE Trans. on PAMI 23(7), 762–766 (2001)
Loog, M., Duin, R.P.W.: Linear Dimensionality Reduction via a Heteroscedastic Extension of LDA: The Chernoff Criterion. IEEE Trans. on PAMI 26(6), 732–739 (2004)
Cover, T.M., Thomas, J.A.: Elements of Information Theory. John Wiley Sons, Inc., Chichester (1991)
Yang, Y., Pedersen, J.O.: A Comparative Study on Feature Selection in Text Categorization. In: ICML 1997 (1997)
Kwak, N., Choi, C.: Input Feature Selection by Mutual Information Based on Parzen Window. IEEE Trans. on PAMI 24(12), 1667–1671 (2002)
Vasconcelos, N.: Feature Selection by Maximum Marginal Diversity. In: NIPS 2002 (2002)
Wu, Y., Zhang, A.: Feature Selection for Classifying High-Dimensional Numerical Data. In: CVPR 2004 (2004)
Peng, H., Long, F., Ding, C.: Feature Selection Based on Mutual Information Criteria of Max-Dependency, Max-Relevance, and Min-Redundancy. IEEE Trans. on PAMI 27(8), 1226–1238 (2005)
Mitra, P., Murthy, C.A., Pal, S.K.: Unsupervised Feature Selection Using Feature Similarity. IEEE Trans. on PAMI 24(3), 301–312 (2002)
Vasconcelos, N., Vasconcelos, M.: Scalable Discriminant Feature Selection for Image Retrieval and Recognition. In: CVPR 2004 (2004)
Torkkola, K., Campbell, W.M.: Mutual Information in Learning Feature Transformations. In: ICML 2000 (2000)
Torkkola, K.: Feature Extraction by Non-Parametric Mutual Information Maximization. J. Machine Learning Research, 1415–1438 (2003)
Figueiredo, M.A.T.: Adaptive Sparseness for Supervised Learning. IEEE Trans. on PAMI 25(9), 1150–1159 (2003)
Krishnapuram, B., Hartemink, A.J., Carin, L., Figueiredo, M.A.T.: A Bayesian Approach to Joint Feature Selection and Classifier Design. IEEE Trans. on PAMI 26(9), 1105–1111 (2004)
Hyvarinen, A., Oja, E.: Independent Component Analysis: Algorithms and Applications. Neural Networks 13(4-5), 411–430 (2000)
Phillips, P.J., Moon, H., Rizvi, S.A., Rauss, P.J.: The FERET Evaluation Methodology for Face-Recognition Algorithms. IEEE Trans. PAMI 22(10), 1090–1104 (2000)
Messer, K., Matas, J., Kittler, J., Luettin, J., Maitre, G.: XM2VTSDB: The Extended M2VTS Database. In: Proc. of Int.l Conf. Audio- and Video-based Person Authentication (1999)
Martinez, A.M., Benavente, R.: The AR Face Database. CVC Technical Report 24, Purdue University (1998)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2006 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Lin, D., Tang, X. (2006). Conditional Infomax Learning: An Integrated Framework for Feature Extraction and Fusion. In: Leonardis, A., Bischof, H., Pinz, A. (eds) Computer Vision – ECCV 2006. ECCV 2006. Lecture Notes in Computer Science, vol 3951. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11744023_6
Download citation
DOI: https://doi.org/10.1007/11744023_6
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-33832-1
Online ISBN: 978-3-540-33833-8
eBook Packages: Computer ScienceComputer Science (R0)