Skip to main content
SpringerLink
Log in
Book cover

International Conference on Artificial Neural Networks

ICANN 2019: Artificial Neural Networks and Machine Learning – ICANN 2019: Theoretical Neural Computation pp 3–19Cite as

Bidirectional Associative Memory with Block Coding: A Comparison of Iterative Retrieval Methods

  • Conference paper
  • First Online:

  • 2971 Accesses

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 11727))

Abstract

Recently, Gripon and Berrou (2011) have investigated a recurrently connected Willshaw-type auto-associative memory with block coding, a particular sparse coding method, reporting a significant increase in storage capacity compared to earlier approaches. In this study we verify and generalize their results by implementing bidirectional hetero-associative networks and comparing the performance of various retrieval methods both with block coding and without block coding. For iterative retrieval in networks of size \(n=4096\) our data confirms that block-coding with the so-called “sum-of-max” strategy performs best in terms of output noise (which is the normalized Hamming distance between stored and retrieved patterns), whereas the information storage capacity of the classical models cannot be exceeded because of the reduced Shannon information of block patterns. Our simulation experiments also provide accurate estimates of the maximum pattern number that can be stored at a tolerated noise level of 1%. It is revealed that block coding is most beneficial for sparse activity where each pattern has only \(k\sim \log n\) active units.

The authors acknowledge support by the state of Baden-Württemberg through bwHPC.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  1. Albus, J.: A theory of cerebellar function. Math. Biosci. 10, 25–61 (1971)

    Article  Google Scholar 

  2. Aliabadi, B.K., Berrou, C., Gripon, V., Jiang, X.: Storing sparse messages in networks of neural cliques. IEEE Trans. Neural Netw. Learn. Syst. 25, 980–989 (2014)

    Article  Google Scholar 

  3. Bentz, H., Hagstroem, M., Palm, G.: Information storage and effective data retrieval in sparse matrices. Neural Netw. 2, 289–293 (1989)

    Article  Google Scholar 

  4. Bogacz, R., Brown, M., Giraud-Carrier, C.: Model of familiarity discrimination in the perirhinal cortex. J. Comput. Neurosci. 10, 5–23 (2001)

    Article  Google Scholar 

  5. Braitenberg, V.: Cell assemblies in the cerebral cortex. In: Heim, R., Palm, G. (eds.) Theoretical Approaches to Complex Systems. Lecture Notes in Biomathematics, vol. 21, pp. 171–188. Springer, Heidelberg (1978). https://doi.org/10.1007/978-3-642-93083-6_9

    Chapter  Google Scholar 

  6. Braitenberg, V., Schüz, A.: Anatomy of the Cortex: Statistics and Geometry. Springer, Berlin (1991). https://doi.org/10.1007/978-3-662-02728-8

    Book  Google Scholar 

  7. Buckingham, J., Willshaw, D.: Performance characteristics of the associative net. Netw.: Comput. Neural Syst. 3, 407–414 (1992)

    Article  Google Scholar 

  8. Chechik, G., Meilijson, I., Ruppin, E.: Synaptic pruning in development: a computational account. Neural Comput. 10(7), 1759–1777 (1998)

    Article  Google Scholar 

  9. Chklovskii, D., Mel, B., Svoboda, K.: Cortical rewiring and information storage. Nature 431, 782–788 (2004)

    Article  Google Scholar 

  10. Cover, T., Thomas, J.: Elements of Information Theory. Wiley, New York (1991)

    Book  Google Scholar 

  11. Dayan, P., Willshaw, D.: Optimising synaptic learning rules in linear associative memory. Biol. Cybern. 65, 253–265 (1991)

    Article  Google Scholar 

  12. Fay, R., Kaufmann, U., Knoblauch, A., Markert, H., Palm, G.: Combining visual attention, object recognition and associative information processing in a neurobotic system. In: Wermter, S., Palm, G., Elshaw, M. (eds.) Biomimetic Neural Learning for Intelligent Robots. LNCS (LNAI), vol. 3575, pp. 118–143. Springer, Heidelberg (2005). https://doi.org/10.1007/11521082_8

    Chapter  Google Scholar 

  13. Ferro, D., Gripon, V., Jiang, X.: Nearest neighbour search using binary neural networks. In: Proceedings of the International Joint Conference on Neural Networks (IJCNN), July 2016

    Google Scholar 

  14. Fransen, E., Lansner, A.: A model of cortical associative memory based on a horizontal network of connected columns. Netw. Comput. Neural Syst. 9, 235–264 (1998)

    Article  Google Scholar 

  15. Gardner, E.: Maximum storage capacity in neural networks. Europhys. Lett. 4, 481–485 (1987)

    Article  Google Scholar 

  16. Gardner-Medwin, A.: The recall of events through the learning of associations between their parts. Proc. Roy. Soc. London Ser. B 194, 375–402 (1976)

    Article  Google Scholar 

  17. Greene, D., Parnas, M., Yao, F.: Multi-index hashing for information retrieval. In: Proceedings of the 35th Annual Symposium on Foundations of Computer Science, pp. 722–731 (1994)

    Google Scholar 

  18. Gripon, V., Berrou, C.: Sparse neural networks with large learning diversity. IEEE Trans. Neural Netw. 22(7), 1087–1096 (2011)

    Article  Google Scholar 

  19. Gripon, V., Berrou, C.: Nearly-optimal associative memories based on distributed constant weight codes. In: Proceedings of the IEEE Information Theory and Applications Workshop (ITA), pp. 269–273 (2012)

    Google Scholar 

  20. Gripon, V., Heusel, J., Löwe, M., Vermet, F.: A comparative study of sparse associative memories. J. Stat. Phys. 164(1), 105–129 (2016)

    Article  MathSciNet  Google Scholar 

  21. Gripon, V., Löwe, M., Vermet, F.: Associative memories to accelerate approximate nearest neighbor search. Appl. Sci. 8(9), 1676 (2018)

    Article  Google Scholar 

  22. Hebb, D.: The Organization of Behavior: A Neuropsychological Theory. Wiley, New York (1949)

    Google Scholar 

  23. Honegger, K., Campbell, R., Turner, G.: Cellular-resolution population imaging reveals robust sparse coding in the Drosophila mushroom body. J. Neurosci. 31(33), 11772–11785 (2011)

    Article  Google Scholar 

  24. Hopfield, J.: Neural networks and physical systems with emergent collective computational abilities. Proc. Natl. Acad. Sci. USA 79, 2554–2558 (1982)

    Article  MathSciNet  Google Scholar 

  25. Huyck, C., Orengo, V.: Information retrieval and categorization using a cell assembly network. Neural Comput. Appl. 14(4), 282–289 (2005)

    Article  Google Scholar 

  26. Johansson, C., Lansner, A.: Imposing biological constraints onto an abstract neocortical attractor network model. Neural Comput. 19(7), 1871–1896 (2007)

    Article  MathSciNet  Google Scholar 

  27. Kanerva, P.: Sparse Distributed Memory. MIT Press, Cambridge (1988)

    MATH  Google Scholar 

  28. Knoblauch, A.: Optimal matrix compression yields storage capacity 1 for binary willshaw associative memory. In: Kaynak, O., Alpaydin, E., Oja, E., Xu, L. (eds.) ICANN/ICONIP - 2003. LNCS, vol. 2714, pp. 325–332. Springer, Heidelberg (2003). https://doi.org/10.1007/3-540-44989-2_39

    Chapter  Google Scholar 

  29. Knoblauch, A.: Neural associative memory with optimal Bayesian learning. Neural Comput. 23(6), 1393–1451 (2011)

    Article  MathSciNet  Google Scholar 

  30. Knoblauch, A.: Efficient associative computation with discrete synapses. Neural Comput. 28(1), 118–186 (2016)

    Article  MathSciNet  Google Scholar 

  31. Knoblauch, A., Körner, E., Körner, U., Sommer, F.: Structural plasticity has high memory capacity and can explain graded amnesia, catastrophic forgetting, and the spacing effect. PLoS One 9(5), e96485 (2014). 1–19

    Article  Google Scholar 

  32. Knoblauch, A., Palm, G.: Iterative retrieval and block coding in auto- and hetero-associative memory. Submitted to Neural Computation (2019)

    Google Scholar 

  33. Knoblauch, A., Palm, G., Sommer, F.: Memory capacities for synaptic and structural plasticity. Neural Comput. 22(2), 289–341 (2010)

    Article  MathSciNet  Google Scholar 

  34. Knoblauch, A., Sommer, F.: Structural plasticity, effectual connectivity, and memory in cortex. Front. Neuroanat. 10(63), 1–20 (2016)

    Google Scholar 

  35. Kohonen, T.: Associative Memory: A System Theoretic Approach. Springer, Heidelberg (1977). https://doi.org/10.1007/978-3-642-96384-1

    Book  MATH  Google Scholar 

  36. Kryzhanovsky, B., Litinskii, L., Mikaelian, A.: Vector-neuron models of associative memory. In: 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541; IJCNN-04), vol. 2, pp. 909–914 (2004)

    Google Scholar 

  37. Lansner, A.: Associative memory models: from the cell-assembly theory to biophysically detailed cortex simulations. Trends Neurosci. 32(3), 178–186 (2009)

    Article  MathSciNet  Google Scholar 

  38. Lansner, A., Ekeberg, O.: A one-layer feedback artificial neural network with a Bayesian learning rule. Int. J. Neural Syst. 1(1), 77–87 (1989)

    Article  Google Scholar 

  39. Lansner, A., Holst, A.: A higher order Bayesian neural network with spiking units. Int. J. Neural Syst. 7(2), 115–128 (1996)

    Article  Google Scholar 

  40. Laurent, G.: Olfactory network dynamics and the coding of multidimensional signals. Nat. Rev. Neurosci. 3, 884–895 (2002)

    Article  Google Scholar 

  41. Marr, D.: A theory of cerebellar cortex. J. Physiol. 202(2), 437–470 (1969)

    Article  Google Scholar 

  42. Marr, D.: Simple memory: a theory for archicortex. Philos. Trans. Roy. Soc. London Ser. B 262, 24–81 (1971)

    Article  Google Scholar 

  43. Mu, X., Artiklar, M., Watta, P., Hassoun, M.: An RCE-based associative memory with application to human face recognition. Neural Process. Let. 23, 257–271 (2006)

    Article  Google Scholar 

  44. Palm, G.: On associative memories. Biol. Cybern. 36, 19–31 (1980)

    Article  Google Scholar 

  45. Palm, G.: Neural Assemblies: An Alternative Approach to Artificial Intelligence. Springer, Berlin (1982). https://doi.org/10.1007/978-3-642-81792-2

    Book  Google Scholar 

  46. Palm, G.: Novelty: Information and Surprise. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-29075-6

    Book  MATH  Google Scholar 

  47. Palm, G.: Neural associative memories and sparse coding. Neural Netw. 37, 165–171 (2013)

    Article  Google Scholar 

  48. Palm, G., Sommer, F.: Information capacity in recurrent McCulloch-Pitts networks with sparsely coded memory states. Network 3, 177–186 (1992)

    Article  Google Scholar 

  49. Palm, G., Sommer, F.: Associative data storage and retrieval in neural nets. In: Domany, E., van Hemmen, J., Schulten, K. (eds.) Models of Neural Networks III, pp. 79–118. Springer, New York (1996). https://doi.org/10.1007/978-1-4612-0723-8_3

    Chapter  Google Scholar 

  50. Prager, R., Fallside, F.: The modified Kanerva model for automatic speech recognition. Comput. Speech Lang. 3, 61–81 (1989)

    Article  Google Scholar 

  51. Pulvermüller, F.: The Neuroscience of Language: On Brain Circuits of Words and Serial Order. Cambridge University Press, Cambridge (2003)

    Book  Google Scholar 

  52. Rehn, M., Sommer, F.: Storing and restoring visual input with collaborative rank coding and associative memory. Neurocomputing 69, 1219–1223 (2006)

    Article  Google Scholar 

  53. Rolls, E.: A theory of hippocampal function in memory. Hippocampus 6, 601–620 (1996)

    Article  Google Scholar 

  54. Sacramento, J., Burnay, F., Wichert, A.: Regarding the temporal requirements of a hierarchical Willshaw network. Neural Netw. 25, 84–93 (2012)

    Article  Google Scholar 

  55. Schwenker, F., Sommer, F., Palm, G.: Iterative retrieval of sparsely coded associative memory patterns. Neural Netw. 9, 445–455 (1996)

    Article  Google Scholar 

  56. Sommer, F., Palm, G.: Improved bidirectional retrieval of sparse patterns stored by Hebbian learning. Neural Netw. 12, 281–297 (1999)

    Article  Google Scholar 

  57. Steinbuch, K.: Die Lernmatrix. Kybernetik 1, 36–45 (1961)

    Article  Google Scholar 

  58. Wichert, A.: Cell assemblies for diagnostic problem-solving. Neurocomputing 69, 810–824 (2006)

    Article  Google Scholar 

  59. Willshaw, D., Buneman, O., Longuet-Higgins, H.: Non-holographic associative memory. Nature 222, 960–962 (1969)

    Article  Google Scholar 

  60. Wu, F.: The potts model. Rev. Mod. Phys. 54, 235–268 (1982)

    Article  MathSciNet  Google Scholar 

  61. Yao, Z., Gripon, V., Rabbat, M.: A GPU-based associative memory using sparse neural networks. In: Proceedings of the IEEE International Conference on High-Performance-Computing and Simulation (HPCS), pp. 688–692 (2014)

    Google Scholar 

Download references

Acknowledgments

The authors are grateful to Friedhelm Schwenker and Fritz Sommer for valuable discussions. The authors acknowledge support by the state of Baden-Württemberg through bwHPC.

Author information

Authors and Affiliations

  1. KEIM Institute, Albstadt-Sigmaringen University, Albstadt, Germany

    Andreas Knoblauch

  2. Institute of Neural Information Processing, Ulm University, Ulm, Germany

    Günther Palm

Authors
  1. Andreas Knoblauch
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Günther Palm
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Andreas Knoblauch .

Editor information

Editors and Affiliations

  1. Helmholtz Zentrum München - Deutsches Forschungszentrum für Gesundheit und Umwelt (GmbH), Neuherberg, Germany

    Igor V. Tetko

  2. Institute of Computer Science, Czech Academy of Sciences, Prague 8, Czech Republic

    Věra Kůrková

  3. Helmholtz Zentrum München - Deutsches Forschungszentrum für Gesundheit und Umwelt (GmbH), Neuherberg, Germany

    Pavel Karpov

  4. Helmholtz Zentrum München - Deutsches Forschungszentrum für Gesundheit und Umwelt (GmbH), Neuherberg, Germany

    Fabian Theis

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Knoblauch, A., Palm, G. (2019). Bidirectional Associative Memory with Block Coding: A Comparison of Iterative Retrieval Methods. In: Tetko, I., Kůrková, V., Karpov, P., Theis, F. (eds) Artificial Neural Networks and Machine Learning – ICANN 2019: Theoretical Neural Computation. ICANN 2019. Lecture Notes in Computer Science(), vol 11727. Springer, Cham. https://doi.org/10.1007/978-3-030-30487-4_1

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-30487-4_1

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-30486-7

  • Online ISBN: 978-3-030-30487-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation