Autonomous Robots

, Volume 43, Issue 8, pp 2163–2182 | Cite as

ART neural network-based integration of episodic memory and semantic memory for task planning for robots

  • Jauwairia NasirEmail author
  • Deok-Hwa Kim
  • Jong-Hwan Kim


Automated task planning for robots faces great challenges in that the sequences of events needed for a particular task are mostly required to be hard-coded. This can be a cumbersome process, especially, when the user wants a robot to learn a large number of similar tasks with different objects that are semantically related. We propose a novel approach of user preference-based integrated multi-memory model (pMM-ART). This approach focuses on exploiting a semantic hierarchy of objects alongside an episodic memory for enhancing the behavior of an autonomous agent. We analyze the functioning principle of the proposed model by teaching it a few distinct domestic tasks and observe that it is able to carry out a large number of similar tasks based on the semantic similarities between learned objects. We also demonstrate, via experiments using Mybot, our ability to reach those goals that are not possible without the integration of semantic knowledge with episodic memory.


Adaptive resonance theory Task planning Cognition Semantic memory Episodic memory User preference 



This work was supported by the National Research Foundation of Korea (NRF) Grant funded by the Korea (MSIP) (No. NRF-2014R1A2A1A10051551) and the Technology Innovation Program, 10045252, funded by the Korea MOTIE. The authors would like to thank Yong-Ho Yoo for his guidance during experiments on Mybot. The authors would also like to thank Jennifer Olsen, a post-doc at Computer–Human Interaction in Learning and Instruction Lab., for her feedback on the draft.

Supplementary material

Supplementary material 1 (mp4 33522 KB)


  1. Al-Moadhen, A., Qiu, R., Packianather, M., Ji, Z., & Setchi, R. (2013). Integrating robot task planner with common-sense knowledge base to improve the efficiency of planning. Procedia Computer Science, 22, 211–220.CrossRefGoogle Scholar
  2. Benjamin, D. P., Lyons, D., & Lonsdale, D. (2004). Adapt: A cognitive architecture for robotics. In Proceedings of the international conference on cognitive modeling (pp. 337–338).Google Scholar
  3. Carpenter, G., Grossberg, S., & Rosen, D. (1991). Fuzzy art: Fast stable learning and categorization of analog patterns by an adaptive resonance system. Neural Networks, 4, 759–771.CrossRefGoogle Scholar
  4. Carpenter, G. A., & Grossberg, S. (1987). A massively parallel architecture for a self-organizing neural pattern recognition machine. Computer Vision, Graphics, and Image Processing, 37, 54–115.CrossRefGoogle Scholar
  5. Cunningham, P. (1998). CBR: Strengths and weaknesses. In Proceedings of the 11th international conference on industrial and engineering applications of artificial intelligence and expert systems: Tasks and methods in applied artificial intelligence (pp. 517–524).CrossRefGoogle Scholar
  6. Dayoub, F., Duckett, T., & Cielniak, G. (2010). Toward an object-based semantic memory for long-term operation of mobile service robots. In IROS workshop on semantic mapping and autonomous knowledge acquisition.Google Scholar
  7. Galindo, C., Fernandez-Madrigal, J., Gonzalez, J., & Saffiotti, A. (2008). Robot task planning using semantic maps. Robotics and Autonomous Systems, 56(11), 955–966.CrossRefGoogle Scholar
  8. Gao, S., & Tan, A. H. (2014). A multi-memory modeling approach. In Proceedings of the international joint conference on neural networks (pp. 1542–1548).Google Scholar
  9. Greenberg, D. L., & Verfaellie, M. (2015). Interdependence of episodic and semantic memory: Evidence from neuropsychology. Journal of the International Neuropsychological Society, 16, 748–753.CrossRefGoogle Scholar
  10. Hawkins, J., George, D., & Niemasik, J. (2009). Sequence memory for prediction, inference and behaviour. Philosophical Transactions of the Royal Society B, 364, 1203–1209.CrossRefGoogle Scholar
  11. Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9, 1735–1780.CrossRefGoogle Scholar
  12. Irish, M., & Piguet, O. (2013). The pivotal role of semantic memory in remembering the past and imagining the future. Frontiers in Behavioral Neuroscience.
  13. Jeong, I. B., Lee, S. J., & Kim, J. H. (2015). RRT\(\ast \)-quick: A motion planning algorithm with faster convergence rate. In J. H. Kim, W. Yang, J. Jo, P. Sincak, & H. Myung (Eds.), Robot intelligence technology and applications 3. Advances in intelligent systems and computing (Vol. 345). Cham: Springer.Google Scholar
  14. Ji, Z., Qiu, R., Noyvirt, A., Soroka, A., Packianather, M., Setchi, R., et al. (2012). Towards automated task planning for service robots using semantic knowledge representation. In Proceedings of the IEEE international conference on industrial information (pp. 1194–1201).Google Scholar
  15. Kim, J. H., Choi, S. H., Park, I. W., & Zaheer, S. A. (2013). Intelligence technology for robots that think. IEEE Computational Intelligence Magazine, 8, 70–84.CrossRefGoogle Scholar
  16. Laird, J. E., Newell, A., & Rosenbloom, P. S. (1987). Soar: An architecture for general intelligence. Artificial Intelligence, 33(1), 1–64. Scholar
  17. Leconte, F., Ferland, F., & Michaud, F. (2015). Design and integration of a spatio-temporal memory with emotional influences to categorize and recall the experiences of an autonomous mobile robot. Autonomous Robots, 40, 831–848.CrossRefGoogle Scholar
  18. Levine, B., Turner, G. R., Tisserand, D., Hevanor, S. J., Graham, S. J., & McIntosh, A. R. (2004). The functional neuroanatomy of episodic and semantic autobiographical remembering: A prospective functional mri study. Journal of Cognitive Neuroscience, 16, 1633–1646.CrossRefGoogle Scholar
  19. McRae, K., & Jones, M. N. (2013). The Oxford handbook of cognitive psychology. Oxford: Oxford University Press (Chap Semantic memory).Google Scholar
  20. Mermillod, M., Bugaiska, A., & Bonin, P. (2013). The stability–plasticity dilemma: Investigating the continuum from catastrophic forgetting to age-limited learning effects. Frontiers in Psychology, 4, 504. Scholar
  21. Nasir, J., & Kim, J. H. (2016). User preference-based integrated multi-memory neural model for improving the cognitive abilities of autonomous robots. Master’s thesis, Korea Advanced Institute of Science and Technology.Google Scholar
  22. Nasir, J., Yoo, Y. H., Kim, D. H., & Kim, J. H. (2018). User preference-based dual-memory neural model with memory consolidation approach. IEEE Transactions on Neural Networks and Learning Systems, 29(6), 2294–2308. Scholar
  23. Nuxoll, A. M., & Laird, J. E. (2007). Extending cognitive architecture with episodic memory. In Proceedings of the 22nd national conference on artificial intelligence AAAI’07(Vol. 2, pp. 1560–1565). New York: AAAI Press.Google Scholar
  24. Nuxoll, A. M., & Laird, J. E. (2012). Enhancing intelligent agents with episodic memory. Cognitive Systems Research, 17–18, 34–48. Scholar
  25. Riesbeck, C. K., & Schank, R. (1989). Inside case-based reasoning. Hillsdale: L. Erlbaum Associates Inc.Google Scholar
  26. Rogers III, J. G., & Christensen, H. I. (2013). Robot planning with a semantic map. In IEEE international conference on robotics and automation (pp. 2239–2244).Google Scholar
  27. Shapiro, S., & Bona, J. P. (2009). The glair cognitive architecture. International Journal of Machine Consciousness, 2(2), 307–332. Scholar
  28. Stachowicz, D., & Kruijff, G. M. (2012). Episodic-like memory for cognitive robots. IEEE Transactions on Autonomous Mental Development, 4(1), 1–16. Scholar
  29. Starzyk, J. A., & He, H. (2007). Anticipation-based temporal sequences learning in hierarchical structure. IEEE Transactions on Neural Networks, 18(2), 344–358. Scholar
  30. Starzyk, J. A., & He, H. (2009). Spatio-temporal memories for machine learning: A long-term memory organization. IEEE Transactions on Neural Networks, 20, 768–780. Scholar
  31. Subagdja, B., & Tan, A. H. (2012). iFALCON: A neural architecture for hierarchical planning. Neurocomputing, 86, 124–139. Scholar
  32. Sucan, I. A., & Chitta, S. (2011). Moveit!
  33. Tan, A. H., Carpenter, G. A., & Grossberg, S. (2007). Intelligence through interaction: Towards a unified theory for learning. In D. Liu, S. Fei, Z. G. Hou, H. Zhang, & C. Sun (Eds.), Advances in neural networks—ISNN 2007 (pp. 1094–1103). Berlin: Springer.CrossRefGoogle Scholar
  34. Tan, A. H., Feng, Y. H., & Ong, Y. S. (2010). A self-organizing neural architecture integrating desire, intention and reinforcement learning. Neurocomputing, 73(7–9), 1465–1477. Scholar
  35. Taylor, S. E., Vineyard, C. M., Healy, M. J., Caudell, T. P., Cohen, N. J., Watson, P., et al. (2009). Memory in silico: Building a neuromimetic episodic cognitive model. In Proceedings of world congress on computer science and information engineering (Vol. 5, pp. 733–737).Google Scholar
  36. Tscherepanow, M. (2010) Topoart: A topology learning hierarchical art network. In Proceedings of the international conference on artificial neural networks (pp. 157–167).Google Scholar
  37. Tscherepanow, M., Kuhnel, S., & Riechers, S. (2012). Episodic clustering of data streams using a topology-learning neural network. In Proceedings of the European conference on artificial intelligence—Workshop on active and incremental learning (pp. 22–24).Google Scholar
  38. Tulving, E. (1972). Episodic and semantic memory. New York: Academic.Google Scholar
  39. Tulving, E. (1983). Elements of episodic memory. New York: Oxford University Press.Google Scholar
  40. Tulving, E. (2002). Episodic memory: From mind to brain. Annual Review of Psychology, 53, 1–25.CrossRefGoogle Scholar
  41. Veiga, T. A., Miraldo, P., Ventura, R., & Lima, P. U. (2016). Efficient object search for mobile robots in dynamic environments: Semantic map as an input for the decision maker. In Proceedings of the IEEE/RSJ international conference on intelligent robots and systems (pp. 2745–2750).Google Scholar
  42. Wang, D., & Arbib, M. A. (1990). Complex temporal sequence learning based on short-term memory. Proceedings of the IEEE, 78(9), 1536–1543. Scholar
  43. Wang, D., & Yuwono, B. (1995). Anticipation-based temporal pattern generation. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 25, 615–628.CrossRefGoogle Scholar
  44. Wang, L. (1999). Multi-associative neural networks and their application to learning and retrieving complex spatio-temporal sequences. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 29, 73–82.CrossRefGoogle Scholar
  45. Wang, W., Subagdja, B., Tan, A. H., & Starzyk, J. A. (2012a). Neural modeling of episodic memory: Encoding, retrieval, and forgetting. IEEE Transactions on Neural Networks and Learning Systems, 23(10), 1574–1586. Scholar
  46. Wang, W., Subagdja, B., Tan, A. H., & Tan, Y. (2012b). A self-organizing multi-memory system for autonomous agents. In Proceedings of the international joint conference on neural networks (pp. 252–258).Google Scholar
  47. Wang, W., Tan, A. H., & Teow, L. (2017). Semantic memory modeling and memory interaction in learning agents. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 47(11), 2882–2895. Scholar
  48. Wu, C., Lenz, I., & Saxena, A. (2014). Hierarchical semantic labeling for task-relevant RGB-D perception. In Robotics: Science and systems.

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.School of Electrical EngineeringKorea Advanced Institute of Science TechnologyDaejeonKorea

Personalised recommendations