Skip to main content
SpringerLink
Log in

Robots that Imagine – Can Hippocampal Replay Be Utilized for Robotic Mnemonics?

  • Conference paper
  • First Online:
Biomimetic and Biohybrid Systems (Living Machines 2019)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 11556))

Included in the following conference series:

Abstract

Neurophysiological studies on hippocampal replay, which was a phenomenon first shown in rodents as the reactivation of previously active hippocampal cells, has shown it to be potentially important for mnemonic functions such as memory consolidation/recall, learning and planning. Since its discovery, a small number of neuronal models have been developed to attempt to describe the workings of this phenomenon. But it may be possible to utilize hippocampal replay to help solve some of the difficult challenges that face robotic cognition, learning and memory, and/or be used for the development of biomimetic robotics. Here we review these models in the hope of learning their workings, and see that their neural network structures may be integrated into current neural network based algorithms for robotic spatial memory, and perhaps are particularly suited for reinforcement learning paradigms.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Ambrose, R.E., Pfeiffer, B.E., Foster, D.J.: Reverse replay of hippocampal place cells is uniquely modulated by changing reward. Neuron 91(5), 1124–1136 (2016)

    Article  Google Scholar 

  2. Ariav, G., Polsky, A., Schiller, J.: Submillisecond precision of the input-output transformation function mediated by fast sodium dendritic spikes in basal dendrites of CA1 pyramidal neurons. J. Neurosci. 23(21), 7750–7758 (2003)

    Article  Google Scholar 

  3. Aubin, L., Khamassi, M., Girard, B.: Prioritized sweeping neural DynaQ with multiple predecessors, and hippocampal replays. In: Vouloutsi, V., et al. (eds.) Living Machines 2018. LNCS (LNAI), vol. 10928, pp. 16–27. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-95972-6_4

    Chapter  Google Scholar 

  4. Braun, E.K., Wimmer, G.E., Shohamy, D.: Retroactive and graded prioritization of memory by reward. Nat. Commun. 9(1), 4886 (2018)

    Article  Google Scholar 

  5. Byrne, P., Becker, S., Burgess, N.: Remembering the past and imagining the future: a neural model of spatial memory and imagery. Psychol. Rev. 114(2), 340 (2007)

    Article  Google Scholar 

  6. Cazé, R., Khamassi, M., Aubin, L., Girard, B.: Hippocampal replays under the scrutiny of reinforcement learning models. J. Neurophysiol. 120(6), 2877–2896 (2018)

    Article  Google Scholar 

  7. Chenkov, N., Sprekeler, H., Kempter, R.: Memory replay in balanced recurrent networks. PLoS Comput. Biol. 13(1), e1005359 (2017)

    Article  Google Scholar 

  8. Cutsuridis, V., Hasselmo, M.: Spatial memory sequence encoding and replay during modeled theta and ripple oscillations. Cogn. Comput. 3(4), 554–574 (2011)

    Article  Google Scholar 

  9. Cutsuridis, V., Taxidis, J.: Deciphering the role of CA1 inhibitory circuits in sharp wave-ripple complexes. Front. Syst. Neurosci. 7, 13 (2013)

    Article  Google Scholar 

  10. Diba, K., Buzsáki, G.: Forward and reverse hippocampal place-cell sequences during ripples. Nat. Neurosci. 10(10), 1241 (2007)

    Article  Google Scholar 

  11. Foster, D.J.: Replay comes of age. Ann. Rev. Neurosci. 40, 581–602 (2017)

    Article  Google Scholar 

  12. Foster, D.J., Wilson, M.A.: Reverse replay of behavioural sequences in hippocampal place cells during the awake state. Nature 440(7084), 680 (2006)

    Article  Google Scholar 

  13. Gerstner, W., Lehmann, M., Liakoni, V., Corneil, D., Brea, J.: Eligibility traces and plasticity on behavioral time scales: experimental support of NeoHebbian three-factor learning rules. Front. Neural Circuits 12, 53 (2018)

    Article  Google Scholar 

  14. Gomperts, S.N., Kloosterman, F., Wilson, M.A.: VTA neurons coordinate with the hippocampal reactivation of spatial experience. Elife 4, e05360 (2015)

    Article  Google Scholar 

  15. Hafting, T., Fyhn, M., Molden, S., Moser, M.B., Moser, E.I.: Microstructure of a spatial map in the entorhinal cortex. Nature 436(7052), 801 (2005)

    Article  Google Scholar 

  16. Haga, T., Fukai, T.: Recurrent network model for learning goal-directed sequences through reverse replay. Elife 7, e34171 (2018)

    Article  Google Scholar 

  17. Humphries, M.D., Prescott, T.J.: The ventral basal ganglia, a selection mechanism at the crossroads of space, strategy, and reward. Prog. Neurobiol. 90(4), 385–417 (2010)

    Article  Google Scholar 

  18. Ikemoto, S., Yang, C., Tan, A.: Basal ganglia circuit loops, dopamine and motivation: a review and enquiry. Behav. Brain Res. 290, 17–31 (2015)

    Article  Google Scholar 

  19. Jahnke, S., Timme, M., Memmesheimer, R.M.: A unified dynamic model for learning, replay, and sharp-wave/ripples. J. Neurosci. 35(49), 16236–16258 (2015)

    Article  Google Scholar 

  20. Jauffret, A., Cuperlier, N., Gaussier, P.: From grid cells and visual place cells to multimodal place cell: a new robotic architecture. Front. Neurorobot. 9, 1 (2015)

    Article  Google Scholar 

  21. Johnson, A., Redish, A.D.: Hippocampal replay contributes to within session learning in a temporal difference reinforcement learning model. Neural Netw. 18(9), 1163–1171 (2005)

    Article  Google Scholar 

  22. Martinez-Hernandez, U., Damianou, A., Camilleri, D., Boorman, L.W., Lawrence, N., Prescott, T.J.: An integrated probabilistic framework for robot perception, learning and memory. In: 2016 IEEE International Conference on Robotics and Biomimetics (ROBIO), pp. 1796–1801. IEEE (2016)

    Google Scholar 

  23. Matheus Gauy, M., et al.: A hippocampal model for behavioral time acquisition and fast bidirectional replay of spatio-temporal memory sequences. Front. Neurosci. 12, 961 (2018)

    Article  Google Scholar 

  24. Mattar, M.G., Daw, N.D.: Prioritized memory access explains planning and hippocampal replay. Nat. Neurosci. 21(11), 1609 (2018)

    Article  Google Scholar 

  25. Milford, M.J., Wyeth, G.F., Prasser, D.: RatSLAM: a hippocampal model for simultaneous localization and mapping. In: Proceedings of the IEEE International Conference on Robotics and Automation, vol. 1, pp. 403–408. IEEE (2004)

    Google Scholar 

  26. Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529 (2015)

    Article  Google Scholar 

  27. Molter, C., Sato, N., Yamaguchi, Y.: Reactivation of behavioral activity during sharp waves: a computational model for two stage hippocampal dynamics. Hippocampus 17(3), 201–209 (2007)

    Article  Google Scholar 

  28. O’Keefe, J., Dostrovsky, J.: The hippocampus as a spatial map: preliminary evidence from unit activity in the freely-moving rat. Brain Res. 34(1), 171–175 (1971)

    Article  Google Scholar 

  29. O’Keefe, J., Recce, M.L.: Phase relationship between hippocampal place units and the EEG theta rhythm. Hippocampus 3(3), 317–330 (1993)

    Article  Google Scholar 

  30. Ólafsdóttir, H.F., Bush, D., Barry, C.: The role of hippocampal replay in memory and planning. Curr. Biol. 28(1), R37–R50 (2018)

    Article  Google Scholar 

  31. Pennartz, C., Lee, E., Verheul, J., Lipa, P., Barnes, C.A., McNaughton, B.: The ventral striatum in off-line processing: ensemble reactivation during sleep and modulation by hippocampal ripples. J. Neurosci. 24(29), 6446–6456 (2004)

    Article  Google Scholar 

  32. Redgrave, P., Prescott, T.J., Gurney, K.: The basal ganglia: a vertebrate solution to the selection problem? Neuroscience 89(4), 1009–1023 (1999)

    Article  Google Scholar 

  33. Richmond, P., Buesing, L., Giugliano, M., Vasilaki, E.: Democratic population decisions result in robust policy-gradient learning: a parametric study with gpu simulations. PLoS ONE 6(5), e18539 (2011)

    Article  Google Scholar 

  34. Saravanan, V., et al.: Transition between encoding and consolidation/replay dynamics via cholinergic modulation of can current: a modeling study. Hippocampus 25(9), 1052–1070 (2015)

    Article  Google Scholar 

  35. Schultz, W.: Predictive reward signal of dopamine neurons. J. Neurophysiol. 80(1), 1–27 (1998)

    Article  MathSciNet  Google Scholar 

  36. Skaggs, W.E., McNaughton, B.L.: Replay of neuronal firing sequences in rat hippocampus during sleep following spatial experience. Science 271(5257), 1870–1873 (1996)

    Article  Google Scholar 

  37. Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT press, Cambridge (2018)

    MATH  Google Scholar 

  38. Taube, J.S.: Head direction cells and the neurophysiological basis for a sense of direction. Prog. Neurobiol. 55(3), 225–256 (1998)

    Article  Google Scholar 

  39. Taxidis, J., Coombes, S., Mason, R., Owen, M.R.: Modeling sharp wave-ripple complexes through a CA3-CA1 network model with chemical synapses. Hippocampus 22(5), 995–1017 (2012)

    Article  Google Scholar 

  40. Trouche, S., et al.: A hippocampus-accumbens tripartite neuronal motif guides appetitive memory in space. Cell 176(6), 1393–1406 (2019)

    Article  Google Scholar 

  41. Vasilaki, E., Frémaux, N., Urbanczik, R., Senn, W., Gerstner, W.: Spike-based reinforcement learning in continuous state and action space: when policy gradient methods fail. PLoS Comput. Biol. 5(12), e1000586 (2009)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, University of Sheffield, Sheffield, S10 2TN, UK

    Matthew T. Whelan, Eleni Vasilaki & Tony J. Prescott

  2. Sheffield Robotics, University of Sheffield, Sheffield, S1 3JD, UK

    Matthew T. Whelan & Tony J. Prescott

Authors
  1. Matthew T. Whelan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Eleni Vasilaki
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tony J. Prescott
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Matthew T. Whelan .

Editor information

Editors and Affiliations

  1. University of Bath, Bath, UK

    Uriel Martinez-Hernandez

  2. SPECS, Institute for Bioengineering of Catalonia, Barcelona, Spain

    Vasiliki Vouloutsi

  3. SPECS, Institute for Bioengineering of Catalonia, Barcelona, Spain

    Anna Mura

  4. University of Sheffield, Sheffield, UK

    Michael Mangan

  5. Osaka University, Suita, Japan

    Minoru Asada

  6. University of Sheffield, Sheffield, UK

    Tony J. Prescott

  7. SPECS, ICREA, BIST, Barcelona, Spain

    Paul F.M.J. Verschure

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Whelan, M.T., Vasilaki, E., Prescott, T.J. (2019). Robots that Imagine – Can Hippocampal Replay Be Utilized for Robotic Mnemonics?. In: Martinez-Hernandez, U., et al. Biomimetic and Biohybrid Systems. Living Machines 2019. Lecture Notes in Computer Science(), vol 11556. Springer, Cham. https://doi.org/10.1007/978-3-030-24741-6_24

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-24741-6_24

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-24740-9

  • Online ISBN: 978-3-030-24741-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation