A Gray-Box Approach for Curriculum Learning

  • Francesco Foglino
  • Matteo Leonetti
  • Simone SagratellaEmail author
  • Ruggiero Seccia
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 991)


Curriculum learning is often employed in deep reinforcement learning to let the agent progress more quickly towards better behaviors. Numerical methods for curriculum learning in the literature provides only initial heuristic solutions, with little to no guarantee on their quality. We define a new gray-box function that, including a suitable scheduling problem, can be effectively used to reformulate the curriculum learning problem. We propose different efficient numerical methods to address this gray-box reformulation. Preliminary numerical results on a benchmark task in the curriculum learning literature show the viability of the proposed approach.


Curriculum learning Reinforcement learning Black-box optimization Scheduling problem 


  1. 1.
    Gpyopt: a bayesian optimization framework in python. (2016)
  2. 2.
    Belotti, P., Kirches, C., Leyffer, S., Linderoth, J., Luedtke, J., Mahajan, A.: Mixed-integer nonlinear optimization. Acta Numer. 22, 1–131 (2013)MathSciNetCrossRefGoogle Scholar
  3. 3.
    Bergstra, J.: Hyperopt: distributed asynchronous hyperparameter optimization in python (2013)Google Scholar
  4. 4.
    Bergstra, J., Yamins, D., Cox, D.D.: Making a science of model search: hyperparameter optimization in hundreds of dimensions for vision architectures (2013)Google Scholar
  5. 5.
    Bergstra, J.S., Bardenet, R., Bengio, Y., Kégl, B.: Algorithms for hyper-parameter optimization. In: Advances in Neural Information Processing Systems, pp. 2546–2554 (2011)Google Scholar
  6. 6.
    Custódio, A.L., Scheinberg, K., Nunes Vicente, L.: Methodologies and software for derivative-free optimization. In: Advances and Trends in Optimization with Engineering Applications, pp. 495–506 (2017)CrossRefGoogle Scholar
  7. 7.
    Di Pillo, G., Liuzzi, G., Lucidi, S., Piccialli, V., Rinaldi, F.: A DIRECT-type approach for derivative-free constrained global optimization. Comput. Optim. Appl. 65(2), 361–397 (2016)MathSciNetCrossRefGoogle Scholar
  8. 8.
    Foglino, F., Leonetti, M.: An optimization framework for task sequencing in curriculum learning (2019). arXiv preprint arXiv:1901.11478
  9. 9.
    Frazier, P.I.: A tutorial on bayesian optimization (2018). arXiv preprint arXiv:1807.02811
  10. 10.
    Leonetti, M., Kormushev, P., Sagratella, S.: Combining local and global direct derivative-free optimization for reinforcement learning. Cybern. Inf. Technol. 12(3), 53–65 (2012)Google Scholar
  11. 11.
    Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A.A., Veness, J., Bellemare, M.G., Graves, A., Riedmiller, M., Fidjeland, A.K., Ostrovski, G., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529 (2015)CrossRefGoogle Scholar
  12. 12.
    Rasmussen, C.E.: Gaussian processes in machine learning. In: Advanced Lectures on Machine Learning, pp. 63–71. Springer (2004)Google Scholar
  13. 13.
    Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., De Freitas, N.: Taking the human out of the loop: a review of bayesian optimization. Proc. IEEE 104(1), 148–175 (2016)CrossRefGoogle Scholar
  14. 14.
    Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. In: Advances in Neural Information Processing Systems, pp. 2951–2959 (2012)Google Scholar
  15. 15.
    Svetlik, M., Leonetti, M., Sinapov, J., Shah, R., Walker, N., Stone, P.: Automatic curriculum graph generation for reinforcement learning agents. In: AAAI, pp. 2590–2596 (2017)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.School of ComputingUniversity of LeedsLeedsUK
  2. 2.Department of Computer, Control and Management Engineering Antonio Ruberti, SapienzaUniversity of RomeRomaItaly

Personalised recommendations