Abstract
Recognizing patterns is an integral aspect of human intelligence; we know that every winter brings cold weather and snowfall. We therefore go to the stores beforehand to purchase coats and tools that will ensure our comfort and survival. We are not shocked when the season changes, instead we learn to manage in each new season. We instilled this ability to detect and cope with seasonality into an autonomous agent- Chippy. Chippy uses a reinforcement learner to gather rewards as it explores its environment. Seasonal changes are constructed into Chippy’s environment by changing rewards at regular intervals. We allowed Chippy to operate at different levels of metacognition and compared the amount of rewards gathered when Chippy operates at each level. Results show that Chippy’s reinforcement learner performs best when Chippy metacognitively monitors not only patterns in expectation violations but also patterns in the suggestions made.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Pastorino, E.E.: What is Psychology? Foundations, Applications and Integration, p. 226. Cengage Learning, Boston (2018)
Oladipupo, T.: Types of machine learning algorithms. In: Zhang, Y. (ed.) New Advances in Machine Learning. IntechOpen, Rijeka (2010). https://doi.org/10.5772/9385
Josyula, D, M’Bale, K.: Bounded metacognition. In: Fifth International Conference on Advanced Cognitive Technologies and Applications, pp. 147–152 (2013)
Fryxell, J.M., Sinclair, A.R.E.: Causes and consequences of migration by large herbivores. Trends Ecol. Evol. 3, 237–241 (1988)
Mysterud, A.: Seasonal migration pattern and home range of roe deer (Capreolus capreolus) in an altitudinal gradient in southern Norway. J. Zool. (Lond.) 247, 479–486 (1999)
Gursky, S.: Effect of seasonality on the behavior of an insectivorous primate, Tarsius spectrum. Int. J. Primatol. 21, 477 (2000)
Langridge, K.V., Broom, M., Osorio, D.: Selective signalling by cuttlefish to predators. Curr. Biol. 17, 1044–1045 (2007). https://doi.org/10.1016/j.cub.2007.10.028
Hultgren, K.M., Stachowicz, J.J.: Size-related habitat shifts facilitated by positive preference induction in a marine kelp crab. Behav. Ecol. 21, 329–336 (2010)
Wright III, D.E.: Finding a temporal frame comparison function for the metacognitive loop. UMBC (Doctoral dissertation) (2011)
Schmill, M., Anderson, M., Fults, S., Josyula, D., Oates, T., Perlis, D., Shahri, H.H., Wilson, S., Wright, D.: The metacognitive loop and reasoning about anomalies. In: Raja, A. (ed.) Metareasoning: Thinking About Thinking. MIT Press, Cambridge (2011)
Anderson, M.L., Oates, T., Chong, W., Perlis, D.: The metacognitive loop I: enhancing reinforcement learning with metacognitive monitoring and control for improved perturbation tolerance. J. Exp. Theor. Artif. Intell. 18(3), 387–411 (2006)
Ng, A.Y., Harada, D., Russell, S.J.: Policy invariance under reward transformations: theory and application to reward shaping. In: Proceedings of the Sixteenth International Conference on Machine Learning, pp. 278–287 (1999)
Tsumori, K., Ozawa, S.: Incremental learning in dynamic environments using neural network with long-term memory. In: Proceedings of the International Conference on Neural Networks, pp. 2583–2588 (2003)
Wiering, M.A.: Reinforcement learning in dynamic environments using instantiated information. In: Proceedings of the Eighteenth International Conference on Machine Learning, vol. 585/592 (2001)
Acknowledgements
We would like to thank Dean Wright for providing us his dissertation code. We would also like to thank Dr. Don Perlis and the Active logic and Metacognitive Computing group at UMD, College Park for discussions on this topic. This work is supported by MAST Collaborative Technology Alliance – Contract No. W911NF-08-2-004.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
David, J., Banks, C., Josyula, D. (2019). Levels of Metacognition and Their Applicability to Reinforcement Learning. In: Samsonovich, A. (eds) Biologically Inspired Cognitive Architectures 2018. BICA 2018. Advances in Intelligent Systems and Computing, vol 848. Springer, Cham. https://doi.org/10.1007/978-3-319-99316-4_9
Download citation
DOI: https://doi.org/10.1007/978-3-319-99316-4_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-99315-7
Online ISBN: 978-3-319-99316-4
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)