Adaptation of NN Complexity to Empirical Information

Abstract

Instrumental observation of a natural phenomenon represents a transmission of information which is generally subject to random disturbances. In the article the scattering of empirical data provided by observation of a certain state is described by a probability distribution function. It is further applied at the estimation of probability distribution of a compound phenomenon which must be characterized by several possible states. The uncertainty of observation is commonly described by the information entropy. It is shown that the empirical information I _e , which is defined as the difference between the entropies of a compound phenomenon and a single state, characterizes a complexity of observed phenomenon. With an increasing number of observations N the value of I _e , increases less quickly as log N and converges to a fixed value 1 _∞. A proper number of empirical samples sufficient to represent the phenomenon can be estimated as K _r = exp I _∞. The redundancy of empirical observations is therefore defined by the excess complexity R = log N — I _∞. When the phenomenon is modeled by a radial basis function NN a proper number of neurons can be described by the parameter K _r . An optimal NN structure can be obtained by minimizing the objective function which is comprised of the redundancy of the model and the information divergence between the representative and empirical distribution.