Abstract
It is known that the number of the edge detectors significantly exceeds that of input signals in the visual system of the brains. This phenomenon has been often regarded as overcomplete independent component analysis (ICA) and some generative models have been proposed. Though the models are effective, they need to assume some ad-hoc prior probabilistic models. Recently, the InfoMin principle was proposed as a comprehensive framework with minimal prior assumptions for explaining the information processing in the brains and its usefulness has been verified in the classic non-overcomplete cases. In this paper, we propose a new ICA contrast function for overcomplete cases, which is deductively derived from the the InfoMin and InfoMax principles without any prior models. Besides, we construct an efficient fixed-point algorithm for optimizing it by an approximate Newton’s method. Numerical experiments verify the effectiveness of the proposed method.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Linsker, R.: Self-organization in a perceptual network. Computer 21(3), 105–117 (1988)
Nadal, J.P., Parga, N.: Nonlinear neurons in the low-noise limit: a factorial code maximizes information-transfer. Network: Computation in Neural Systems 5(4), 565–581 (1994)
Bell, A.J., Sejnowski, T.J.: The independent components of natural scenes are edge filters. Vision Research 37(23), 3327–3338 (1997)
Bell, A.J., Sejnowski, T.J.: An information-maximization approach to blind separation and blind deconvolution. Neural Computation 7, 1129–1159 (1995)
van Hateren, J.H., van der Schaaf, A.: Independent component filters of natural images compared with simple cells in primary visual cortex. Proceedings of the Royal Society of London: B 265, 359–366 (1998)
Hyvärinen, A., Hoyer, P.O., Inki, M.: Topographic independent component analysis. Neural Computation 13(7), 1527–1558 (2001)
Hoyer, P.O., Hyvärinen, A.: A multi-layer sparse coding network learns contour coding from natural images. Vision Research 42(12), 1593–1605 (2002)
Karklin, Y., Lewicki, M.S.: Learning higher-order structures in natural images. Network: Computation in Neural Systems 14, 483–499 (2003)
Matsuda, Y., Yamaguchi, K.: The InfoMin criterion: an information theoretic unifying objective function for topographic mappings. In: Kaynak, O., Alpaydın, E., Oja, E., Xu, L. (eds.) ICANN 2003 and ICONIP 2003. LNCS, vol. 2714, pp. 401–408. Springer, Heidelberg (2003)
Matsuda, Y., Yamaguchi, K.: The InfoMin principle for ICA and topographic mappings. In: Rosca, J.P., Erdogmus, D., Príncipe, J.C., Haykin, S. (eds.) ICA 2006. LNCS, vol. 3889, pp. 958–965. Springer, Heidelberg (2006)
Matsuda, Y., Yamaguchi, K.: A fixed-point algorithm of topographic ICA. In: Kollias, S.D., Stafylopatis, A., Duch, W., Oja, E. (eds.) ICANN 2006. LNCS, vol. 4132, pp. 587–594. Springer, Heidelberg (2006)
Matsuda, Y., Yamaguchi, K.: A connection-limited neural network by infomax and infomin. In: Proceedings of IJCNN 2008, Hong Kong. IEEE, Los Alamitos (in press, 2008)
Lewicki, M.S., Sejnowski, T.J.: Learning overcomplete representations. Neural Computation 12(2), 337–365 (2000)
Hyvärinen, A., Hoyer, P.O., Inki, M.: Estimating overcomplete independent component bases for image windows. Journal of Mathematical Imaging and Vision 17, 139–152 (2002)
Hyvärinen, A.: Fast and robust fixed-point algorithms for independent component analysis. IEEE Transactions on Neural Networks 10(3), 626–634 (1999)
Amari, S., Cichocki, A.: A new learning algorithm for blind signal separation. In: Touretzky, D., Mozer, M., Hasselmo, M. (eds.) Advances in Neural Information Processing Systems, vol. 8, pp. 757–763. MIT Press, Cambridge (1996)
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2008 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Matsuda, Y., Yamaguchi, K. (2008). An Overcomplete ICA Algorithm by InfoMax and InfoMin. In: Kůrková, V., Neruda, R., Koutník, J. (eds) Artificial Neural Networks - ICANN 2008. ICANN 2008. Lecture Notes in Computer Science, vol 5163. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-87536-9_15
Download citation
DOI: https://doi.org/10.1007/978-3-540-87536-9_15
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-87535-2
Online ISBN: 978-3-540-87536-9
eBook Packages: Computer ScienceComputer Science (R0)