, Volume 3, Issue 3, pp 128–135

Adaptive exploration through covariance matrix adaptation enables developmental motor learning

Research Article

DOI: 10.2478/s13230-013-0108-6

Cite this article as:
Stulp, F. & Oudeyer, PY. Paladyn (2012) 3: 128. doi:10.2478/s13230-013-0108-6


The “Policy Improvement with Path Integrals” (PI2) [25] and “Covariance Matrix Adaptation — Evolutionary Strategy” [8] are considered to be state-of-the-art in direct reinforcement learning and stochastic optimization respectively. We have recently shown that incorporating covariance matrix adaptation into PI2- which yields the PICMA2 algorithm — enables adaptive exploration by continually and autonomously reconsidering the exploration/exploitation trade-off. In this article, we provide an overview of our recent work on covariance matrix adaptation for direct reinforcement learning [22–24], highlight its relevance to developmental robotics, and conduct further experiments to analyze the results. We investigate two complementary phenomena from developmental robotics. First, we demonstrate PICMA2’s ability to adapt to slowly or abruptly changing tasks due to its continual and adaptive exploration. This is an important component of life-long skill learning in dynamic environments. Second, we show on a reaching task how PICMA2 subsequently releases degrees of freedom from proximal to more distal limbs as learning progresses. A similar effect is observed in human development, where it is known as ‘proximodistal maturation’.


reinforcement learning covariance matrix adaptation developmental robotics adaptive exploration proximodistal maturation 

Copyright information

© Versita Warsaw and Springer-Verlag Wien 2012

Authors and Affiliations

  1. 1.Robotics and Computer VisionENSTA-ParisTechParisFrance
  2. 2.FLOWERS TeamINRIA Bordeaux Sud-OuestTalenceFrance

Personalised recommendations