Structural Information Control for Flexible Competitive Learning
In this paper, we propose a new information theoretic method called structural information to overcome fundamental problems inherent in conventional competitive learning such as dead neurons and deciding on the appropriate number of neurons in the competitive layer. Our method is based on defining and controlling several kinds of information, thus generating particular neuron firing patterns. For one firing pattern, some neurons are completely inactive, meaning that some dead neurons are generated. For another firing pattern, all neurons are active, that is, there is no dead neurons. This means that we can control the number of dead neurons and choose the appropriate number of neurons by controlling information content. We applied this method to simple pattern classification to show that information can be controlled, and that different neuron firing patterns can be generated.
KeywordsStructural Information Input Pattern Firing Pattern Order Information Input Unit
Unable to display preview. Download preview PDF.
- D. E. Rumelhart D. Zipser, “Feature discovery by competitive learning” in Parallel Distributed Processing (D. E. Rumelhart G. E. Hinton R. J. Williams, eds., vol. 1, pp. 151–193, Cambrige: MIT Press, 1986.Google Scholar
- D. Diesieno, “Adding a conscience to competitive learning” in Proceedings of IEEE International Conference on Neural Networks, pp. 117–124, IEEE, 1988.Google Scholar
- A. Luk S. Lien, “Properties of the generalized lotto-type competitive learning” in Proceedings of International conference on neural information processing, pp. 1180–1185, 2000.Google Scholar
- L. L. Gatlin, Information Theory and Living Systems. Columbia University Press, 1972.Google Scholar