Abstract
Autonomous reinforcement learning (RL) robots usually need to learn from raw, high dimensional data generated by visual sensors and often corrupted by noise. These sorts of tasks are quite challenging and cannot be addressed without an efficient mechanism to encode and simplify the raw data. A recent study proposed an artificial curios robot (ACR) for this problem. However, this model is incapable of handling non-Markovian tasks and discovering spatiotemporal patterns in its milieu. This paper presents a method to solve this problem by extending ACR. A straightforward, but not efficient, solution is to keep recoding of previous observations which makes the algorithm intractable. We, instead, construct a perceptual context in a compact way. Using different environments, we show that the proposed algorithm can discover the regularity in its environment without any prior information on the task.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Lin, L.-J., Mitchell, T.M.: Reinforcement learning with hidden states. In: Proc. 2nd Int. Conf. From Animals to Animats, pp. 271–280 (1993)
Chrisman, L.: Reinforcement learning with perceptual aliasing: The perceptual distinctions approach. In: Proc. 10th Nat. Conf. AI, pp. 183–188. AAAI Press (1992)
Bakker, B.: Reinforcement Learning with Long Short-Term Memory. In: Neural Information Processing Systems (NIPS), pp. 1475–1482 (2001)
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (1998)
Kaelbling, L.P., Littman, M.L., Moore, A.W.: Reinforcement Learning: A Survey. Journal of Artificial Intelligence Research 4, 237–285 (1996)
Sutton, R.S.: Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In: Proc. 7th Int. Conf. Machine Learning, pp. 216–224 (1990)
Vigorito, C.M., Barto, A.G.: Intrinsically Motivated Hierarchical Skill Learning in Structured Environments. IEEE Trans. Auton. Mental Develop. 2, 132–143 (2010)
Barto, A.G., Singh, S., Chentanez, N.: Intrinsically motivated learning of hierarchical collections of skills. In: Proc. 3rd Int. Conf. Development and Learning, La Jolla, CA, pp. 112–119 (2004)
Schmidhuber, J.: A possibility for implementing curiosity and boredom in model-building neural controllers. In: Meyer, J.A., Wilson, S.W. (eds.) Proc. Int. Conf. Simulation of Adaptive Behavior: From Animals to Animats, pp. 222–227. MIT Press (1991)
Oudeyer, P.-Y., Kaplan, F.: What is intrinsic motivation? A typology of computational approaches. Frontiers in Neurorobotics 1 (2007)
Lopes, M., Montesano, L.: Active Learning for Autonomous Intelligent Agents: Exploration, Curiosity, and Interaction. arXiv preprint arXiv:1403.1497v1 (2014)
Frank, M., Leitner, J., Stollenga, M., Förster, A., Schmidhuber, J.: Curiosity driven reinforcement learning for motion planning on humanoids. Frontiers in Neurorobotics 7 (2014)
Ngo, H., Luciw, M., Förster, A., Schmidhuber, J.: Confidence-based progress-driven self-generated goals for skill acquisition in developmental robots. Frontiers in Psychology 4 (2013)
Luciw, M., Graziano, V., Ring, M., Schmidhuber, J.: Artificial curiosity with planning for autonomous perceptual and cognitive development. In: 2011 IEEE Int. Conf. on Development and Learning (ICDL), pp. 1–8. IEEE Press (2011)
Asada, M., Hosoda, K., Kuniyoshi, Y., Ishiguro, H., Inui, T., Yoshikawa, Y., Ogino, M., Yoshida, C.: Cognitive Developmental Robotics: A Survey. IEEE Trans. Auton. Mental Develop. 1, 12–34 (2009)
Hruby, R., Maas, L.M., Fedor-Freybergh, P.G.: Early brain development toward shaping of human mind: an integrative psychoneurodevelopmental model in prenatal and perinatal medicine. Neuro. Endocrinol. Lett. 34, 447–463 (2013)
Andreakis, A., Hoyningen-Huene, N.v., Beetz, M.: Incremental unsupervised time series analysis using merge growing neural gas. In: Príncipe, J.C., Miikkulainen, R. (eds.) WSOM 2009. LNCS, vol. 5629, pp. 10–18. Springer, Heidelberg (2009)
Carpenter, G.A., Grossberg, S., Rosen, D.B.: Fuzzy ART: Fast stable learning and categorization of analog patterns by an adaptive resonance system. Neural Networks 4, 759–771 (1991)
Weng, J., Luciw, M.: Dually optimal neuronal layers: Lobe component analysis. IEEE Trans. Auton. Mental Develop. 1, 68–85 (2009)
Schmidhuber, J.: Formal Theory of Creativity, Fun, and Intrinsic Motivation. IEEE Trans. Auton. Mental Develop. 2, 230–247 (2010)
Singh, S.P., Sutton, R.S.: Reinforcement learning with replacing eligibility traces. Machine Learning 22, 123–158 (1996)
Lagoudakis, M.G., Parr, R.: Least-squares policy iteration. The Journal of Machine Learning Research 4, 1107–1149 (2003)
Lange, S., Riedmiller, M.: Deep auto-encoder neural networks in reinforcement learning. In: The 2010 Int. Joint Conf. Neural Networks (IJCNN), pp. 1–8. IEEE Press (2010)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this paper
Cite this paper
Kalhor, D., Loo, C.K. (2014). Artificial Curiosity Driven Robots with Spatiotemporal Regularity Discovery Ability. In: Huang, DS., Jo, KH., Wang, L. (eds) Intelligent Computing Methodologies. ICIC 2014. Lecture Notes in Computer Science(), vol 8589. Springer, Cham. https://doi.org/10.1007/978-3-319-09339-0_9
Download citation
DOI: https://doi.org/10.1007/978-3-319-09339-0_9
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-09338-3
Online ISBN: 978-3-319-09339-0
eBook Packages: Computer ScienceComputer Science (R0)