Advertisement

Associative Data Storage and Retrieval in Neural Networks

  • Günther Palm
  • Friedrich T. Sommer
Part of the Physics of Neural Networks book series (NEURAL NETWORKS)

Synopsis

Associative storage and retrieval of binary random patterns in various neural net models with one-step threshold-detection retrieval and local learning rules are the subject of this chapter. For different heteroas-sociation and autoassociation memory tasks specified by the properties of the pattern sets to be stored and upper bounds on the retrieval errors, we compare the performance of various models of finite as well as asymptotically infinite sizes. In infinite models, we consider the case of asymptotically sparse patterns, where the mean activity in a pattern vanishes, and study two asymptotic fidelity requirements: constant error probabilities and vanishing error probabilities. A signal-to-noise ratio analysis is carried out for one retrieval step where the calculations are comparatively straightforward and easy. As performance measures we propose and evaluate information capacities in bits/synapse which also take into account the important property of fault tolerance. For autoassociation we compare one-step and fixed-point retrieval that is analyzed in the literature by methods of statistical mechanics.

Keywords

Input Pattern Associative Memory Sparse Pattern Optimal Rule Content Pattern 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. [1]
    Hodgkin, A.L., Huxley, A.F. (1952) A quantitative description of membrane current and its application to conduction and excitation in nerve. J. Physiol. (Lond.) 117:500–544Google Scholar
  2. [2]
    McCulloch, W.S., Pitts, W. (1943) A logical calculus of the ideas immanent in neural activity. Bull. Math. Biophys. 5Google Scholar
  3. [3]
    Steinbuch, K (1936) Die Lernmatrix. Kybernetik 1:36CrossRefGoogle Scholar
  4. [4]
    Willshaw, D.J., Buneman, O.P., Longuet-Higgans, H.C. (1969) Nonholographic associative memory. Nature (London) 222:960–962ADSCrossRefGoogle Scholar
  5. [5]
    Rosenblatt, F. (1962) Principle of Neurodynamics (Spartan Books, New York)Google Scholar
  6. [6]
    Little, W.A. (1974) The existence of persistent states in the brain. Math. Biosci. 19:101–120MATHCrossRefGoogle Scholar
  7. [7]
    Kirkpatrick, S., Sherrington, D. (1978) Infinite-ranged models of spin-glasses. Phys. Rev. B 17:4384–4403ADSCrossRefGoogle Scholar
  8. [8]
    Amit, D.J., Gutfreund, H., Sompolinsky, H. (1987) Statistical mechanics of neural networks near saturation. Ann. Phys. 173:30–67ADSCrossRefGoogle Scholar
  9. [9]
    Domany, E., van Hemmen, J.L., Schulten, K. (1991) Models of Neural Networks (Springer-Verlag, Berlin)MATHCrossRefGoogle Scholar
  10. [10]
    Amit, D. J. (1989) Modelling Brain Function (Cambridge University Press, Cambridge)Google Scholar
  11. [11]
    Hertz, J., Krogh, A., Palmer, R. G. (1991) Introduction to the Theory of Neural Computation (Addison Wesley, Redwood City, CA)Google Scholar
  12. [12]
    Uttley, A.M. (1956) Conditional probability machines and conditional reflexes. In: An. Math. Studies 34, Shannon, C.E., McCarthy, J. (Eds.) (Princeton Univ. Press, Princeton, NJ), pp. 237–252Google Scholar
  13. [13]
    Longuett-Higgins, H.C., Willshaw, D.J., Buneman, O.P. (1970) Theories of associative recall. Q. Rev. Biophys. 3:223–244CrossRefGoogle Scholar
  14. [14]
    Amari, S.I. (1971) Characteristics of randomly connected threshold-element networks and network systems. Proc. IEEE 59:35–47MathSciNetCrossRefGoogle Scholar
  15. [15]
    Gardner-Medwin, A.R. (1976) The recall of events through the learning of associations between their parts. Proc. R. Soc. Lond. B. 194:375–402ADSCrossRefGoogle Scholar
  16. [16]
    Kohonen, T. (1977) Associative Memory (Springer-Verlag, Berlin)MATHCrossRefGoogle Scholar
  17. [17]
    Caianiello, E.R. (1961) Outline of a theory of thought processes and thinking machines. J. Theor. Biol. 1:204–225MathSciNetCrossRefGoogle Scholar
  18. [18]
    Holden, A.V. (1976) Models of the Stochastic Activity of Neurons (Springer-Verlag, Berlin)CrossRefGoogle Scholar
  19. [19]
    Abeles, M. (1982) Local Cortical Circuits (Springer-Verlag, Berlin)CrossRefGoogle Scholar
  20. [20]
    Buhmann, J., Schulten, K. (1986) Associative recognition and storage in a model network of physiological neurons. Biol. Cybern. 54:319–335MATHCrossRefGoogle Scholar
  21. [21]
    Anderson, J.A. (1968) A memory storage model utilizing spatial correlation functions. Kybernetik 5:113–119CrossRefGoogle Scholar
  22. [22]
    Anderson, J.A. (1972) A simple neural network generating an interactive memory. Math. Biosci. 14:197–220MATHCrossRefGoogle Scholar
  23. [23]
    Palm, G. (1980) On associative memory. Biol. Cybern. 36:19–31MATHCrossRefGoogle Scholar
  24. [24]
    Nadal, J.-P., Toulouse, G. (1990) Information storage in sparsely coded memory nets. Network 1:61–74MathSciNetMATHCrossRefGoogle Scholar
  25. [25]
    Hopfield, J. J. (1982) Neural networks and physical systems with emergent collective computational abilities. Proc. Natl. Sci. 79:2554–2558MathSciNetADSCrossRefGoogle Scholar
  26. [26]
    van Hemmen, J.L. (1987) Nonlinear networks near saturation. Phys. Rev. A: Math. Gen. 36:1959–1962ADSCrossRefGoogle Scholar
  27. [27]
    Tsodyks, M.V., Feigelman, M.V. (1988) The enhanced storage capacity in neural networks with low activity level. Europhys. Lett. 6:101–105ADSCrossRefGoogle Scholar
  28. [28]
    Amari, S.I. (1989) Statistical neurodynamics of associative memory. Neural Networks 1:63–73CrossRefGoogle Scholar
  29. [29]
    Fontanari, J.F., Köberle, R. (1988) Information processing in synchronous neural networks. J. Phys.France 49:13–23CrossRefGoogle Scholar
  30. [30]
    Palm, G., Sommer, F. T. (1992) Information capacity in recurrent McCulloch-Pitts networks with sparsely coded memory states. Network 3:1–10CrossRefGoogle Scholar
  31. [31]
    Gibson, W.G., Robinson, J. (1992) Statistical analysis of the dynamics of a sparse associative memory. Neural Networks 5:645–662CrossRefGoogle Scholar
  32. [32]
    Hebb, D.O. (1949) The Organization of Behavior (Wiley, New York)Google Scholar
  33. [33]
    Herz, A., Sulzer, B., Kühn, R., van Hemmen, J.L. (1988) The Hebb rule: Storing static and dynamic objects in an associative neural network. Europhys. Lett. 7:663–669; (1989) Hebbian learning reconsidered: Representation of static and dynamic objects in associative neural nets. Biol. Cybern. 60:457-467ADSCrossRefGoogle Scholar
  34. [34]
    Personnaz, L., Dreyfus, G., Toulouse, G. (1986) A biologically constrained learning mechanism in networks of formal neurons. J. Stat. Phys. 43:411–422MathSciNetADSMATHCrossRefGoogle Scholar
  35. [35]
    Personnaz, L., Guyon, I., Dreyfus, G. (1986) Collective computational properties of neural networks: New learning mechanisms. Phys. Rev. A: Math. Gen. 34:4217–4228MathSciNetADSCrossRefGoogle Scholar
  36. [36]
    Palm, G. (1982) Neural Assemblies (Springer-Verlag, Berlin)CrossRefGoogle Scholar
  37. [37]
    Willshaw, D.J., Dayan, P. (1990) Optimal plasticity from matrix memories: What goes up must come down. Neural Comp. 2:85–93CrossRefGoogle Scholar
  38. [38]
    Barto, A.G., Sutton, R.S., Brouwer, P.S. (1981) Associative search network: A reinforcement learning associative memory. Biol. Cybern. 40:201–211MATHCrossRefGoogle Scholar
  39. [39]
    Lamperti, J. (1966) Probability (Benjamin, New York)MATHGoogle Scholar
  40. [40]
    Shannon, C, Weaver, W. (1949) The Mathematical Theory of Communication (University of Illinois Press, Urbana, IL)MATHGoogle Scholar
  41. [41]
    Palm, G. (1992) On the information storage capacity of local learning rules. Neural Comp. 4:703–711CrossRefGoogle Scholar
  42. [42]
    Gardner, E. (1987) Maximum storage capacity in neural networks. Europhys. Lett. 4:481–485ADSCrossRefGoogle Scholar
  43. [43]
    Gardner, E. (1988) The space of interactions in neural network models. J. Phys. A: Math. Gen. 21:257–270ADSCrossRefGoogle Scholar
  44. [44]
    Schwenker, F., Sommer, F.T., Palm, G. (1993) Iterative retrieval of sparsely coded patterns in associative memory. Neuronet′93 Prague Google Scholar
  45. [45]
    Sommer, F.T. (1993) Theorie neuronaler Assoziativspeicher; Lokales Lernen und iteratives Retrieval von Information. Ph.D. thesis, DüsseldorfGoogle Scholar
  46. [46]
    Palm, G., Schwenker, F., Sommer, F.T. (1993) Associative memory and sparse similarity perserving codes. In: From Statistics to Neural Networks: Theory and Pattern Recognition Applications, Cherkassky, V. (Ed.) (Springer NATO ASI Series F) (Springer-Verlag, New York)Google Scholar
  47. [47]
    Palm, G. (1990) Local learning rules and sparse coding in neural networks. In: Advanced Neural Computers, Eckmiller, R. (Ed.) (Elsevier, Amsterdam), pp. 145–150Google Scholar
  48. [48]
    Horner, H. (1989) Neural networks with low levels of activity: Ising vs. McCulloch-Pitts neurons. Z. Phys. B 75:133–136ADSCrossRefGoogle Scholar
  49. [49]
    Willshaw, D.J., Dayan, P. (1991) Optimizing synaptic learning rules in linear associative memories. Biol. Cybern. 50:253–265Google Scholar
  50. [50]
    Peretto, P. (1988) On learning rules and memory storage abilities. J. Phys. France 49:711–726MathSciNetCrossRefGoogle Scholar
  51. [51]
    Palm, G. (1991) Memory capacities of local rules for synaptic modification. Concepts in Neuroscience 2:97–128Google Scholar
  52. [52]
    Horner, H., Bormann, D., Frick, M., Kinzelbach, H., Schmidt, A. (1989) Transients and basins of attraction in neural network models. Z. Phys. B 76:381–398ADSCrossRefGoogle Scholar
  53. [53]
    Buhmann, J., Divko, R., Schulten, K. (1989) Associative memory with high information content. Phys. Rev. A 39:2689–2692MathSciNetADSCrossRefGoogle Scholar
  54. [54]
    Palm, G. (1987) Computing with neural networks. Science 235:1227–1228ADSCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 1996

Authors and Affiliations

  • Günther Palm
    • 1
  • Friedrich T. Sommer
    • 2
  1. 1.Abteilung Neuroinformatik, Fakultät für InformatikUniversität Ulm, Oberer EselsbergUlmGermany
  2. 2.Institut für Medizinische Psychologic und Verhaltensneurobiologic der Universität TübingenTübingenGermany

Personalised recommendations