Advertisement

A Linear Online Guided Policy Search Algorithm

  • Biao Sun
  • Fangzhou Xiong
  • Zhiyong LiuEmail author
  • Xu Yang
  • Hong Qiao
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10638)

Abstract

In reinforcement learning (RL), the guided policy search (GPS), a variant of policy search method, can encode the policy directly as well as search for optimal solutions in the policy space. Even though this algorithm is provided with asymptotic local convergence guarantees, it can not work in a online way for conducting tasks in complex environments since it is trained with a batch manner which requires that all of the training samples should be given at the same time. In this paper, we propose an online version for GPS algorithm, which can learn policies incrementally without complete knowledge of initial positions for training. The experiments witness its efficacy on handling sequentially arriving training samples in a peg insertion task.

Keywords

Reinforcement learning Policy search Online learning 

Notes

Acknowledgments

This work is partly supported by NSFC grants 61375005, U1613213, 61702516, 61210009, MOST grants 2015BAK35B00, 2015BAK35B01, Guangdong Science and Technology Department grant 2016B090910001.

References

  1. 1.
    Kalakrishnan, M., Righetti, L., Pastor, P., Schaal, S.: Learning force control policies for compliant robotic manipulation. In: Proceedings of the 29th International Conference on Machine Learning (2012)Google Scholar
  2. 2.
    Levine, S., Finn, C., Darrell, T., Abbeel, P.: End-to-end training of deep visuomotor policies. J. Mach. Learn. Res. 17(39), 1–40 (2016)MathSciNetzbMATHGoogle Scholar
  3. 3.
    Endo, G., Morimoto, J., Matsubara, T., Nakanishi, J., Cheng, G.: Learning CPG-based biped locomotion with a policy gradient method: application to a humanoid robot. Int. J. Robot. Res. 27(2), 213–228 (2008)CrossRefGoogle Scholar
  4. 4.
    Deisenroth, M.P., Neumann, G., Peters, J., et al.: A survey on policy search for robotics. Found. Trends Robot. 2(1–2), 1–142 (2013)Google Scholar
  5. 5.
    Levine, S., Koltun, V.: Guided policy search. In: Proceedings of the 30th International Conference on Machine Learning, pp. 1–9 (2013)Google Scholar
  6. 6.
    Levine, S., Abbeel, P.: Learning neural network policies with guided policy search under unknown dynamics. In: Advances in Neural Information Processing Systems, pp. 1071–1079 (2014)Google Scholar
  7. 7.
    Levine, S., Koltun, V.: Variational policy search via trajectory optimization. In: Advances in Neural Information Processing Systems, pp. 207–215 (2013)Google Scholar
  8. 8.
    Montgomery, W.H., Levine, S.: Guided policy search via approximate mirror descent. In: Advances in Neural Information Processing Systems, pp. 4008–4016 (2016)Google Scholar
  9. 9.
    Sutton, R.S., Koop, A., Silver, D.: On the role of tracking in stationary environments. In: Proceedings of the 24th international conference on Machine learning, pp. 871–878 (2007)Google Scholar
  10. 10.
    Ruvolo, P., Eaton, E.: ELLA: An efficient lifelong learning algorithm. In: Proceedings of the 30th International Conference on Machine Learning, pp. 507–515 (2013)Google Scholar
  11. 11.
    Chebotar, Y., Kalakrishnan, M., Yahya, A., Li, A., Schaal, S., Levine, S.: Path integral guided policy search. In: International Conference on Robotics and Automation, pp. 3381–3388 (2017)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Biao Sun
    • 1
    • 2
  • Fangzhou Xiong
    • 2
    • 3
  • Zhiyong Liu
    • 2
    • 3
    • 4
    • 5
    Email author
  • Xu Yang
    • 2
  • Hong Qiao
    • 1
    • 2
    • 3
    • 4
    • 5
  1. 1.University of Science and Technology BeijingBeijingChina
  2. 2.The State Key Lab of Management and Control for Complex SystemsInstitute of Automation, Chinese Academy of ScienceBeijingChina
  3. 3.School of Computer and ControlUniversity of Chinese Academy of Sciences (UCAS)BeijingChina
  4. 4.CAS Centre for Excellence in Brain Science and Intelligence Technology (CEBSIT)ShanghaiChina
  5. 5.Cloud Computing CenterChinese Academy of SciencesDongGuanChina

Personalised recommendations