Advertisement

Adaptive Adjacency Kanerva Coding for Memory-Constrained Reinforcement Learning

  • Wei LiEmail author
  • Waleed Meleis
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10934)

Abstract

When encountering continuous, or very large domains, using a compact representation of the state space is preferable for practical reinforcement learning (RL). This approach can reduce the size of the state space and enable generalization by relating similar or neighboring states. However, many state abstraction techniques cannot achieve satisfactory approximation quality in the presence of limited memory resources, while expert state space shaping can be costly and usually does not scale well. We have investigated the principle of Sparse Distributed Memories (SDMs) and applied it as a function approximator to learn good policies for RL. This paper describes a new approach, adaptive adjacency in SDMs, that is capable of representing very large continuous state spaces with a very small collection of prototype states. This algorithm enhances an SDMs architecture to allow on-line, dynamically-adjusting generalization to assigned memory resources to provide high-quality approximation. The memory size and memory allocation no longer need to be manually assigned before and during RL. Based on our results, this approach performs well both in terms of approximation quality and memory usage. The superior performance of this approach over existing SDMs and tile coding (CMACs) is demonstrated through a comprehensive simulation study in two classic domains, Mountain Car with 2 dimensions and Hunter-Prey with 5 dimensions. Our empirical evaluations demonstrate that the adaptive adjacency approach can be used to efficiently approximate value functions with limited memories, and that the approach scales well across tested domains with continuous, large-scale state spaces.

Keywords

Reinforcement learning Function approximation Kanerva coding Dynamic generalization 

References

  1. 1.
    Allen, M., Fritzsche, P.: Reinforcement learning with adaptive Kanerva coding for Xpilot game AI. In: 2011 IEEE Congress of Evolutionary Computation (CEC), pp. 1521–1528. IEEE (2011).  https://doi.org/10.1109/CEC.2011.5949796
  2. 2.
    Chernova, S., Veloso, M.: Tree-based policy learning in continuous domains through teaching by demonstration. In: Proceedings of Workshop on Modeling Others from Observations (MOO 2006) (2006)Google Scholar
  3. 3.
    Chiariotti, F., D’Aronco, S., Toni, L., Frossard, P.: Online learning adaptation strategy for dash clients. In: Proceedings of the 7th International Conference on Multimedia Systems, p. 8. ACM (2016).  https://doi.org/10.1145/2910017.2910603
  4. 4.
    Forbes, J.R.N.: Reinforcement Learning for Autonomous Vehicles. University of California, Berkeley (2002)Google Scholar
  5. 5.
    Frommberger, L.: Qualitative Spatial Abstraction in Reinforcement Learning. Springer, Heidelberg (2010).  https://doi.org/10.1007/978-3-642-16590-0CrossRefGoogle Scholar
  6. 6.
    Geist, M., Pietquin, O.: Algorithmic survey of parametric value function approximation. IEEE Trans. Neural Netw. Learn. Syst. 24(6), 845–867 (2013).  https://doi.org/10.1109/TNNLS.2013.2247418CrossRefGoogle Scholar
  7. 7.
    Hausknecht, M., Khandelwal, P., Miikkulainen, R., Stone, P.: HyperNEAT-GGP: a hyperNEAT-based atari general game player. In: Proceedings of the 14th Annual Conference on Genetic and Evolutionary Computation, pp. 217–224. ACM (2012).  https://doi.org/10.1145/2330163.2330195
  8. 8.
    Kanerva, P.: Sparse distributed memory and related models, vol. 92. NASA Ames Research Center, Research Institute for Advanced Computer Science (1992)Google Scholar
  9. 9.
    Keller, P.W., Mannor, S., Precup, D.: Automatic basis function construction for approximate dynamic programming and reinforcement learning. In: Proceedings of International Conference on Machine Learning (2006).  https://doi.org/10.1145/1143844.1143901
  10. 10.
    Li, L., Baker, T.E., White, S.R., Burke, K.: Pure density functional for strong correlations and the thermodynamic limit from machine learning. Phys. Rev. B 94(24), 245129 (2016)CrossRefGoogle Scholar
  11. 11.
    Li, W., Zhou, F., Meleis, W., Chowdhury, K.: Learning-based and data-driven TCP design for memory-constrained iot. In: 2016 International Conference on Distributed Computing in Sensor Systems (DCOSS), pp. 199–205. IEEE (2016).  https://doi.org/10.1109/DCOSS.2016.8
  12. 12.
    Lin, S., Wright, R.: Evolutionary tile coding: an automated state abstraction algorithm for reinforcement learning. In: Abstraction, Reformulation, and Approximation (2010)Google Scholar
  13. 13.
    Mao, H., Netravali, R., Alizadeh, M.: Neural adaptive video streaming with pensieve. In: Proceedings of the Conference of the ACM Special Interest Group on Data Communication, pp. 197–210. ACM (2017).  https://doi.org/10.1145/3098822.3098843
  14. 14.
    Munos, R., Moore, A.: Variable resolution discretization in optimal control. Mach. Learn. 49(2), 291–323 (2002)CrossRefGoogle Scholar
  15. 15.
    Ratitch, B., Precup, D.: Sparse distributed memories for on-line value-based reinforcement learning. In: Boulicaut, J.-F., Esposito, F., Giannotti, F., Pedreschi, D. (eds.) ECML 2004. LNCS (LNAI), vol. 3201, pp. 347–358. Springer, Heidelberg (2004).  https://doi.org/10.1007/978-3-540-30115-8_33CrossRefzbMATHGoogle Scholar
  16. 16.
    Santamaría, J.C., Sutton, R.S., Ram, A.: Experiments with reinforcement learning in problems with continuous state and action spaces. Adapt. Behav. 6(2), 163–217 (1997)CrossRefGoogle Scholar
  17. 17.
    Smart, W.D., Kaelbling, L.P.: Practical reinforcement learning in continuous spaces. In: ICML, pp. 903–910 (2000)Google Scholar
  18. 18.
    Sutton, R., Barto, A.: Reinforcement Learning: An Introduction. Bradford Books (1998)Google Scholar
  19. 19.
    Whiteson, S., Taylor, M.E., Stone, P., et al.: Adaptive tile coding for value function approximation. University of Texas at Austin, Computer Science Department (2007)Google Scholar
  20. 20.
    Wu, C., Li, W., Meleis, W.: Rough sets-based prototype optimization in Kanerva-based function approximation. In: 2015 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT), vol. 2, pp. 283–291. IEEE (2015).  https://doi.org/10.1109/WI-IAT.2015.179
  21. 21.
    Wu, C., Meleis, W.: Adaptive Kanerva-based function approximation for multi-agent systems. In: Proceedings of the 7th International Joint Conference on Autonomous Agents and Multiagent Systems, vol. 3. pp. 1361–1364. International Foundation for Autonomous Agents and Multiagent Systems (2008).  https://doi.org/10.1145/1402821.1402872

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Northeastern UniversityBostonUSA

Personalised recommendations