Advertisement

Associative Memory Networks

  • Ke-Lin DuEmail author
  • M. N. S. Swamy
Chapter

Abstract

In the brain, knowledge is learnt by associating different types of sensory data. Associative memory is a fundamental function of human brain. It can be realized with neural networks with backward connections. Neural networks for associate memory and their learning algorithms are introduced in this chapter.

References

  1. 1.
    Abu-Mostafa, Y., & St Jacques, J. (1985). Information capability of the Hopfield network. IEEE Transactions on Information Theory, 31(4), 461–464.zbMATHCrossRefGoogle Scholar
  2. 2.
    Aiyer, S. V. B., Niranjan, N., & Fallside, F. (1990). A theoretical investigation into the performance of the Hopfield model. IEEE Transactions on Neural Networks, 1(2), 204–215.CrossRefGoogle Scholar
  3. 3.
    Amari, S. I. (1972). Learning patterns and pattern sequences by self-organizing nets of threshold elements. IEEE Transactions on Computers, 21, 1197–1206.MathSciNetzbMATHCrossRefGoogle Scholar
  4. 4.
    Amiri, M., Saeb, S., Yazdanpanah, M. J., & Seyyedsalehi, S. A. (2008). Analysis of the dynamical behavior of a feedback auto-associative memory. Neurocomputing, 71, 486–494.CrossRefGoogle Scholar
  5. 5.
    Amit, D. J., Gutfreund, G., & Sompolinsky, H. (1987). Statistical mechanics of neural networks near saturation. Annals of Physics, 173, 30–67.CrossRefGoogle Scholar
  6. 6.
    Amit, D. J. (1989). Modeling brain function: The world of attractor neural networks. Cambridge: Cambridge University Press.zbMATHCrossRefGoogle Scholar
  7. 7.
    Amit, D. J., Campbell, C., & Wong, K. Y. M. (1989). The interaction space of neural networks with sign-constrained synapses. Journal of Physics A: General Physics, 22, 4687–4693.MathSciNetzbMATHCrossRefGoogle Scholar
  8. 8.
    Anderson, J. A. (1972). A simple neural network generating interactive memory. Mathematical Biosciences, 14, 197–220.zbMATHCrossRefGoogle Scholar
  9. 9.
    Anderson, J. A., Silverstein, J. W., Ritz, S. A., & Jones, R. S. (1977). Distinctive features, categorical perception, and probability learning: Some applications of a neural model. Psychological Review, 84, 413–451.CrossRefGoogle Scholar
  10. 10.
    Baird, B. (1990). Associative memory in a simple model of oscillating cortex. In D. S. Touretzky (Ed.), Advances in neural information processing systems (Vol. 2, pp. 68–75). San Mateo: Morgan Kaufmann.Google Scholar
  11. 11.
    Bogacz, R., & Brown, M. W. (2003). Comparison of computational models of familiarity discrimination in the perirhinal cortex. Hippocampus, 13, 494–524.CrossRefGoogle Scholar
  12. 12.
    Bruck, J., & Roychowdhury, W. P. (1990). On the number of spurious memories in the Hopfield model. IEEE Transactions on Information Theory, 36(2), 393–397.MathSciNetzbMATHCrossRefGoogle Scholar
  13. 13.
    Casali, D., Costantini, G., Perfetti, R., & Ricci, E. (2006). Associative memory design using support vector machines. IEEE Transactions on Neural Networks, 17(5), 1165–1174.CrossRefGoogle Scholar
  14. 14.
    Chartier, S., & Proulx, R. (2005). NDRAM: Nonlinear dynamic recurrent associative memory for learning bipolar and non-bipolar correlated patterns. IEEE Transactions on Neural Networks, 16, 1393–1400.CrossRefGoogle Scholar
  15. 15.
    Chartier, S., & Boukadoum, M. (2006). A bidirectional heteroassociative memory for binary and grey-level patterns. IEEE Transactions on Neural Networks, 17(2), 385–396.CrossRefGoogle Scholar
  16. 16.
    Chartier, S., Boukadoum, M., & Amiri, M. (2009). BAM learning of nonlinearly separable tasks by using an asymmetrical output function and reinforcement learning. IEEE Transactions on Neural Networks, 20(8), 1281–1292.CrossRefGoogle Scholar
  17. 17.
    Chiueh, T. D., & Goodman, R. M. (1991). Recurrent correlation associative memories. IEEE Transactions on Neural Networks, 2(2), 275–284.CrossRefGoogle Scholar
  18. 18.
    Chiueh, T. D., & Tsai, H. K. (1993). Multivalued associative memories based on recurrent networks. IEEE Transactions on Neural Networks, 4(2), 364–366.CrossRefGoogle Scholar
  19. 19.
    Cohen, M. A., & Grossberg, S. (1983). Absolute stability of global pattern formation and parallel memory storage by competitive neural networks. IEEE Transactions on Systems, Man, and Cybernetics, 13, 815–826.MathSciNetzbMATHCrossRefGoogle Scholar
  20. 20.
    Conway, M. A. (2008). Exploring episodic memory. Handbook of behavioral neuroscience (Vol. 18, pp. 19–29). Amsterdam: Elsevier.Google Scholar
  21. 21.
    Coombes, S., & Taylor, J. G. (1994). Using generalized principal component analysis to achieve associative memory in a Hopfield net. Network, 5, 75–88.zbMATHCrossRefGoogle Scholar
  22. 22.
    Coombes, S., & Campbell, C. (1996). Efficient learning beyond saturation by single-layered neural networks. Technical report 96.6, Bristol Center for Applied Nonlinear Mathematics, University of Bristol, UK.Google Scholar
  23. 23.
    Cortes, J. M., Greve, A., Barrett, A. B., & van Rossum, M. C. W. (2010). Dynamics and robustness of familiarity memory. Neural Computation, 22, 448–466.zbMATHCrossRefGoogle Scholar
  24. 24.
    Cruz, B., Sossa, H., & Barron, R. (2007). A new two-level associative memory for efficient pattern restoration. Neural Processing Letters, 25, 1–16.CrossRefGoogle Scholar
  25. 25.
    Delbem, A. C. B., Correa, L. G., & Zhao, L. (2009). Design of associative memories using cellular neural networks. Neurocomputing, 72, 2180–2188.CrossRefGoogle Scholar
  26. 26.
    Eom, T., Choi, C., & Lee, J. (2002). Generalized asymmetrical bidirectional associative memory for multiple association. Applied Mathematics and Computation, 127, 221–233.MathSciNetzbMATHCrossRefGoogle Scholar
  27. 27.
    Fleisher, M. (1988). The Hopfield model with multi-level neurons. In D. Z. Anderson (Ed.), Neural information processing systems (pp. 278–289). New York: American Institute Physics.Google Scholar
  28. 28.
    Fortney, K., Tweed, D. B., & Sejnowski, T. (2012). Computational advantages of reverberating loops for sensorimotor learning. Neural Computation, 24(3), 611–634.zbMATHCrossRefGoogle Scholar
  29. 29.
    Gardner, E. (1987). Maximum storage capacity in neural networks. Europhysics Letters, 4, 481–485.CrossRefGoogle Scholar
  30. 30.
    Gardner, E. (1988). The space of the interactions in neural network models. Journal of Physics A, 21, 257–270.MathSciNetzbMATHCrossRefGoogle Scholar
  31. 31.
    Grassi, G. (2001). On discrete-time cellular neural networks for associative memories. IEEE Transactions on Circuits and Systems, 48(1), 107–111.zbMATHCrossRefGoogle Scholar
  32. 32.
    Grossberg, S. (1988). Nonlinear neural networks: Principles, mechanisms, and architectures. Neural Networks, 1, 17–61.CrossRefGoogle Scholar
  33. 33.
    Hassoun, M. H., & Watta, P. B. (1996). The Hamming associative memory and its relation to the exponential capacity DAM. In Proceedings of IEEE International Conference on Neural Networks (Vol. 1, pp. 583–587). Washington, DC.Google Scholar
  34. 34.
    Helie, S. (2008). Energy minimization in the nonlinear dynamic recurrent associative memory. Neural Networks, 21, 1041–1044.zbMATHCrossRefGoogle Scholar
  35. 35.
    Hopfield, J. J. (1982). Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences of the USA, 79, 2554–2558.MathSciNetzbMATHCrossRefGoogle Scholar
  36. 36.
    Hui, S., & Zak, S. H. (1992). Dynamic analysis of the brain-state-in-a-box (BSB) neural models. IEEE Transactions on Neural Networks, 3, 86–100.CrossRefGoogle Scholar
  37. 37.
    Ikeda, N., Watta, P., Artiklar, M., & Hassoun, M. H. (2001). A two-level Hamming network for high performance associative memory. Neural Networks, 14, 1189–1200.CrossRefGoogle Scholar
  38. 38.
    Jagota, A., & Mandziuk, J. (1998). Experimental study of Perceptron-type local learning rule for Hopfield associative memory. Information Sciences, 111, 65–81.CrossRefGoogle Scholar
  39. 39.
    Jankowski, S., Lozowski, A., & Zurada, J. M. (1996). Complex-valued multi-state neural associative memory. IEEE Transactions on Neural Networks, 7(6), 1491–1496.CrossRefGoogle Scholar
  40. 40.
    Kamp, Y., & Hasler, M. (1990). Recursive neural networks for associative memory. New York: Wiley.zbMATHGoogle Scholar
  41. 41.
    Kobayashi, K. (1991). On the capacity of a neuron with a non-monotone output function. Network, 2, 237–243.MathSciNetCrossRefGoogle Scholar
  42. 42.
    Kohonen, T. (1972). Correlation matrix memories. IEEE Transactions on Computers, 21(4), 353–359.zbMATHCrossRefGoogle Scholar
  43. 43.
    Kohonen, T. (1989). Self-organization and associative memory. Berlin: Springer.zbMATHCrossRefGoogle Scholar
  44. 44.
    Kosko, B. (1987). Adaptive bidirectional associative memories. Applied Optics, 26, 4947–4960.CrossRefGoogle Scholar
  45. 45.
    Kosko, B. (1988). Bidirectional associative memories. IEEE Transactions on Systems, Man, and Cybernetics, 18(1), 49–60.MathSciNetCrossRefGoogle Scholar
  46. 46.
    Kumaran, D., Hassabis, D., & McClelland, J. L. (2016). What learning systems do intelligent agents need? Complementary learning systems theory updated. Trends in Cognitive Sciences, 20(7), 512–534.CrossRefGoogle Scholar
  47. 47.
    Lee, D. L. (2001). Improving the capacity of complex-valued neural networks with a modified gradient descent learning rule. IEEE Transactions on Neural Networks, 12(2), 439–443.CrossRefGoogle Scholar
  48. 48.
    Leung, C. S., & Chan, L. W. (1997). The behavior of forgetting learning in bidirectional associative memory. Neural Computation, 9, 385–401.zbMATHCrossRefGoogle Scholar
  49. 49.
    Li, J. H., Michel, A. N., & Parod, W. (1989). Analysis and synthesis of a class of neural networks: Linear systems operating on a closed hypercube. IEEE Transactions on Circuits and Systems, 36(11), 1405–1422.MathSciNetzbMATHCrossRefGoogle Scholar
  50. 50.
    Lin, W., & Chen, G. (2009). Large memory capacity in chaotic artificial neural networks: A view of the anti-integrable limit. IEEE Transactions on Neural Networks, 20(8), 1340–1351.CrossRefGoogle Scholar
  51. 51.
    Lippman, R. P. (1987). An introduction to computing with neural nets. IEEE ASSP Magazine, 4(2), 4–22.MathSciNetCrossRefGoogle Scholar
  52. 52.
    Liu, D., & Michel, A. N. (1994). Sparsely interconnected neural networks for associative memories with applications to cellular neural networks. IEEE Transactions on Circuits and Systems, 41, 295–307.zbMATHCrossRefGoogle Scholar
  53. 53.
    Liu, D., & Lu, Z. (1997). A new synthesis approach for feedback neural networks based on the perceptron training algorithm. IEEE Transactions on Neural Networks, 8(6), 1468–1482.CrossRefGoogle Scholar
  54. 54.
    Liu, X.-G., Martin, R. R., Wu, M., & Tang, M.-L. (2008). Global exponential stability of bidirectional associative memory neural networks with time delays. IEEE Transactions on Neural Networks, 19(3), 397–407.CrossRefGoogle Scholar
  55. 55.
    Ma, J. (1997). The stability of the generalized Hopfield networks in randomly asynchronous mode. Neural Networks, 10, 1109–1116.CrossRefGoogle Scholar
  56. 56.
    Ma, J. (1999). The asymptotic memory capacity of the generalized Hopfield network. Neural Networks, 12, 1207–1212.CrossRefGoogle Scholar
  57. 57.
    McClelland, J. L., McNaughton, B. L., & O’Reilly, R. C. (1995). Why there are complementary learning systems in the hippocampus and neocortex: Insights from the successes and failures of connectionist models of learning and memory. Psychological Review, 102, 419–457.CrossRefGoogle Scholar
  58. 58.
    McEliece, R. J., Posner, E. C., Rodemich, E. R., & Venkatesh, S. S. (1987). The capacity of the Hopfield associative memory. IEEE Transactions on Information Theory, 33(4), 461–482.MathSciNetzbMATHCrossRefGoogle Scholar
  59. 59.
    Michel, A. N., Si, J., & Yen, G. (1991). Analysis and synthesis of a class of discrete-time neural networks described on hypercubes. IEEE Transactions on Neural Networks, 2(1), 32–46.CrossRefGoogle Scholar
  60. 60.
    Morita, M. (1993). Associative memory with nonmonotonicity dynamics. Neural Networks, 6, 115–126.CrossRefGoogle Scholar
  61. 61.
    Muezzinoglu, M. K., Guzelis, C., & Zurada, J. M. (2003). A new design method for the complex-valued multistate Hopfield associative memory. IEEE Transactions on Neural Networks, 14(4), 891–899.CrossRefGoogle Scholar
  62. 62.
    Nakagawa, M. (1996). A parameter controlled chaos neural network. Journal of the Physical Society of Japan, 65, 1859–1867.CrossRefGoogle Scholar
  63. 63.
    Nemoto, I., & Kubono, M. (1996). Complex associative memory. Neural Networks, 9(2), 253–261.CrossRefGoogle Scholar
  64. 64.
    Oja, E. (1982). A simplified neuron model as a principal component analyzer. Journal of Mathematical Biology, 15, 267–273.MathSciNetzbMATHCrossRefGoogle Scholar
  65. 65.
    Park, J., & Park, Y. (2000). An optimization approach to design of generalized BSB neural associative memories. Neural Computation, 12, 1449–1462.CrossRefGoogle Scholar
  66. 66.
    Park, J., Kim, H. Y., & Lee, S. W. (2001). A synthesis procedure for associative memories based on space-varying cellular neural networks. Neural Networks, 14, 107–113.CrossRefGoogle Scholar
  67. 67.
    Personnaz, L., Guyon, I., & Dreyfus, G. (1986). Collective computational properties of neural networks: New learning mechanism. Physical Review A, 34(5), 4217–4228.MathSciNetCrossRefGoogle Scholar
  68. 68.
    Ritter, G. X., Sussner, P., & de Leon, J. L. D. (1998). Morphological associative memories. IEEE Transactions on Neural Networks, 9(2), 281–293.CrossRefGoogle Scholar
  69. 69.
    Ritter, G. X., de Leon, J. L. D., & Sussner, P. (1999). Morphological bidirectional associative memories. Neural Networks, 6(12), 851–867.CrossRefGoogle Scholar
  70. 70.
    Rojas, R. (1996). Neural networks: A systematic introduction. Berlin: Springer.zbMATHCrossRefGoogle Scholar
  71. 71.
    Shi, H., Zhao, Y., & Zhuang, X. (1998). A general model for bidirectional associative memories. IEEE Transactions on Systems, Man, and Cybernetics Part B, 28(4), 511–519.CrossRefGoogle Scholar
  72. 72.
    Si, J., & Michel, A. N. (1995). Analysis and synthesis of a class of discrete-time neural networks with multilevel threshold neurons. IEEE Transactions on Neural Networks, 6(1), 105–116.CrossRefGoogle Scholar
  73. 73.
    Storkey, A. J. (1997). Increasing the capacity of the Hopfield network without sacrificing functionality. In W. Gerstner, A. Germond, M. Hastler, & J. Nicoud (Eds.), Proceedings of International Conference on Artificial Neural Networks (ICANN), LNCS (Vol. 1327, pp. 451–456). Berlin: Springer.Google Scholar
  74. 74.
    Storkey, A. J., & Valabregue, R. (1997). Hopfield learning rule with high capacity storage of time-correlated patterns. Electronics Letters, 33(21), 1803–1804.CrossRefGoogle Scholar
  75. 75.
    Sussner, P., & Valle, M. E. (2006). Gray-scale morphological associative memories. IEEE Transactions on Neural Networks, 17(3), 559–570.CrossRefGoogle Scholar
  76. 76.
    Szolgay, P., Szatmari, I., & Laszlo, K. (1997). A fast fixed-point learning method to implement associative memoryon CNNs. IEEE Transactions on Circuits and Systems, 44(4), 362–366.CrossRefGoogle Scholar
  77. 77.
    Taylor, W. (1956). Eletrical simulation of some nervous system functional activities. Information Theory, 3, 314–328.Google Scholar
  78. 78.
    Valle, M. E. (2014). Complex-valued recurrent correlation neural networks. IEEE Transactions on Neural Networks and Learning Systems, 25(9), 1600–1612.CrossRefGoogle Scholar
  79. 79.
    Venkatesh, S. S., & Psaltis, D. (1989). Linear and logarithmic capacities in associative memory. IEEE Transactions on Information Theory, 35, 558–568.MathSciNetCrossRefGoogle Scholar
  80. 80.
    Ventura, D., & Martinez, T. (2000). Quantum associative memory. Information Sciences, 124, 273–296.MathSciNetCrossRefGoogle Scholar
  81. 81.
    Wu, Y., & Batalama, S. N. (2000). An efficient learning algorithm for associative memories. IEEE Transactions on Neural Networks, 11(5), 1058–1066.CrossRefGoogle Scholar
  82. 82.
    Xu, Z. B., Leung, Y., & He, X. W. (1994). Asymmetric bidirectional associative memories. IEEE Transactions on Systems, Man, and Cybernetics, 24(10), 1558–1564.CrossRefGoogle Scholar
  83. 83.
    Yoshizawa, S., Morita, M., & Amari, S. I. (1993). Capacity of associative memory using a nonmonotonic neuron model. Neural Networks, 6, 167–176.CrossRefGoogle Scholar
  84. 84.
    Zeng, Z., & Wang, J. (2008). Design and analysis of high-capacity associative memories based on a class of discrete-time recurrent neural networks. IEEE Transactions on Systems, Man, and Cybernetics Part B, 38(6), 1525–1536.CrossRefGoogle Scholar
  85. 85.
    Zeng, Z., & Wang, J. (2009). Associative memories based on continuous-time cellular neural networks designed using space-invariant cloning templates. Neural Networks, 22, 651–657.zbMATHCrossRefGoogle Scholar

Copyright information

© Springer-Verlag London Ltd., part of Springer Nature 2019

Authors and Affiliations

  1. 1.Department of Electrical and Computer EngineeringConcordia UniversityMontrealCanada
  2. 2.Xonlink Inc.HangzhouChina

Personalised recommendations