Entropy and Learning

Abstract

Recently, neural networks have attracted a lot of attention from the scientific community, as is witnessed by the large number of conferences on the subject, and the introduction of a dedicated journal (neural networks). It has become a truely interdisciplinary field, involving artificial intelligence, neurophysiology, psychology and statistical physics. One of the reasons for this success is the relative high performance of neural networks in a number of search and learning problems (e.g. the construction of associative memories[[1]], optimization[[2]], and classification[[3]]). These tasks are performed rather easily by the human brain, despite its relatively low processing speed (the time for a neuron to fire is in the range of msec, while it takes a fraction of a second to recognize a face). The widely accepted explanation for this fact is the high level of parallel processing in the brain, each neuron being connected to thousands of other neurons. This high connectivity is also the characteristic feature of what are called neural networks.