Advertisement

Regularization in DQN for Parameter-Varying Control Learning Tasks

  • Dazi LiEmail author
  • Chengjia Lei
  • Qibing Jin
  • Min Han
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11555)

Abstract

As an important technique of preventing overfitting, regularization is widely used in supervised learning. However, regularization has not been systematically studied in deep reinforcement learning (deep RL). In this paper, we study the generalization of deep Q-network (DQN), applying with mainstream regularization approaches, including l1, l2 and dropout. We pay attention on agent’s performance not only in original environments, but also in parameter-varying environments which are variational but the same task type. Furthermore, the dropout is modified to make it more adaptive to DQN. Then, a new dropout is proposed to speed up the optimization of DQN. Experiments show that regularization helps deep RL achieve better performance in both original and parameter-varying environments when the number of samples is insufficient.

Keywords

Regularization Deep RL Control learning task 

Notes

Acknowledgements

This work is supported by the National Natural Science Foundation of China (61873022, 61573052), the Beijing Natural Science Foundation (4182045), the China Postdoctoral Science Foundation (2018M640049) and the Fundamental Research Funds for the Central Universities (XK1802-4, ZY1839).

References

  1. 1.
    LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)Google Scholar
  2. 2.
    Hardt, M., Recht, B., Singer, Y.: Train faster, generalize better: stability of stochastic gradient descent. In: 33rd International Conference on Machine Learning, pp. 1868–1877. IMLS Press, New York (2016)Google Scholar
  3. 3.
    Bishop, C.M.: Training with noise is equivalent to Tikhonov regularization. Neural Comput. 7(1), 108–116 (1995)Google Scholar
  4. 4.
    Chandra, B., Sharma, R.K.: Adaptive noise schedule for denoising autoencoder. In: Loo, C.K., Yap, K.S., Wong, K.W., Teoh, A., Huang, K. (eds.) ICONIP 2014. LNCS, vol. 8834, pp. 535–542. Springer, Cham (2014).  https://doi.org/10.1007/978-3-319-12637-1_67Google Scholar
  5. 5.
    Silver, D., et al.: Mastering the game of go without human knowledge. Nature 550, 354–359 (2017)Google Scholar
  6. 6.
    Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT, Cambridge (1998)Google Scholar
  7. 7.
    Farahmand, A.M., Ghavamzadeh, M., Szepesvari, C., Mannor, S.: Regularized policy iteration. In: 22nd Annual Conference on Neural Information Processing Systems, pp. 441–448. Curran Associates Press, Vancouver (2009)Google Scholar
  8. 8.
    Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., Meger, D.: Deep reinforcement learning that matters. arXiv preprint arXiv:1709.06560 (2017)
  9. 9.
    Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. JMLR 15(1), 1929–1958 (2014)Google Scholar
  10. 10.
    Strehl, A.L., Li, L., Littman, M.L.: Reinforcement learning in finite MDPs: PAC analysis. JMLR 10, 2413–2444 (2009)Google Scholar
  11. 11.
    Smith, S.L., Le, Q.V.: Understanding generalization and stochastic gradient descent. arXiv preprint arXiv:1710.06451v1 (2017)
  12. 12.
    Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518, 529–533 (2015)Google Scholar
  13. 13.
    Whiteson, S., Tanner, B., Taylor, M.E., Stone, P.: Protecting against evaluation overfitting in empirical reinforcement learning. In: 2011 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning, Paris, pp. 120–127. IEEE Press (2011)Google Scholar
  14. 14.
    Bouthillier, X., Konda, K., Vincent, P., Memisevic, R.: Dropout as data augmentation. arXiv preprint arXiv:1506.08700 (2016)
  15. 15.
    Gal, Y., Ghahramani, Z.: Dropout as a Bayesian approximation: representing model uncertainty in deep learning. arXiv preprint arXiv:1506.02142 (2015)
  16. 16.
    Riedmiller, M.: Neural fitted Q iteration – first experiences with a data efficient neural reinforcement learning method. In: Gama, J., Camacho, R., Brazdil, P.B., Jorge, A.M., Torgo, L. (eds.) ECML 2005. LNCS, vol. 3720, pp. 317–328. Springer, Heidelberg (2005).  https://doi.org/10.1007/11564096_32Google Scholar
  17. 17.
    Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Zaremba, W.: OpenAI gym. arXiv preprint arXiv:1606.01540 (2016)
  18. 18.
    Hahnloser, R.H.R., Sarpeshkar, R., Mahowald, M.A., Douglas, R.J., Seung, H.S.: Digital selection and analogue amplification coexist in a cortex-inspired silicon circuit. Nature 405, 947–951 (2000)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.College of Information Science and TechnologyBeijing University of Chemical TechnologyBeijingChina
  2. 2.Dalian University of TechnologyDalianChina

Personalised recommendations