Learning Agents with Prioritization and Parameter Noise in Continuous State and Action Space

  • Rajesh MangannavarEmail author
  • Gopalakrishnan Srinivasaraghavan
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11554)


Among the many variants of RL, an important class of problems is where the state and action spaces are continuous—autonomous robots, autonomous vehicles, optimal control are all examples of such problems that can lend themselves naturally to reinforcement based algorithms, and have continuous state and action spaces. In this paper, we introduce a prioritized form of a combination of state-of-the-art approaches such as Deep Q-learning (DQN) and Deep Deterministic Policy Gradient (DDPG) to outperform the earlier results for continuous state and action space problems. Our experiments also involve the use of parameter noise during training resulting in more robust deep RL models outperforming the earlier results significantly. We believe these results are a valuable addition for continuous state and action space problems.


Reinforcement learning Policy search Prioritized learning Parameter noise RL Deep learning Mujoco Policy gradient DDPG 


  1. 1.
    Li, Y.: Deep reinforcement learning: an overview. CoRR. abs/1701.07274 (2017)Google Scholar
  2. 2.
    Doya, K.: Reinforcement learning in continuous time and space. Neural Comput. 12, 219–245 (2000)Google Scholar
  3. 3.
    LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015)Google Scholar
  4. 4.
    Arulkumaran, K., Deisenroth, M.P., Brundage, M., Bharath, A.A.: A brief survey of deep reinforcement learning. CoRR. abs/1708.05866 (2017)Google Scholar
  5. 5.
    Mnih, V., et al.: Playing atari with deep reinforcement learning. CoRR. abs/1312.5602 (2013)Google Scholar
  6. 6.
    Schaul, T., Quan, J., Antonoglou, I., Silver, D.: Prioritized experience replay. CoRR. abs/1511.05952 (2015)Google Scholar
  7. 7.
    Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., Riedmiller, M.: Deterministic policy gradient algorithms. In: Proceedings of the 31st International Conference on International Conference on Machine Learning, Beijing (2014)Google Scholar
  8. 8.
    Konda, V.R., Tsitsiklis, J.N.: Actor-critic algorithms. In: Advances in Neural Information Processing Systems (2000)Google Scholar
  9. 9.
    Feinberg, E.A., Shwartz, A.: Handbook of Markov Decision Processes: Methods and Applications. Springer, Heidelberg (2012). Scholar
  10. 10.
    Lillicrap, T.P., et al.: Continuous control with deep reinforcement learning. CoRR. abs/1509.02971 (2015)Google Scholar
  11. 11.
    Plappert, M., et al.: Parameter space noise for exploration. CoRR. abs/1706.01905 (2017)Google Scholar
  12. 12.
    Schulman, J., Levine, S., Moritz, P., Jordan, M.I., Abbeel, P.: Trust region policy optimization. CoRR. abs/1502.05477 (2015)Google Scholar
  13. 13.
    Todorov, E., Erez, T., Tassa, Y.: MuJoCo: a physics engine for model-based control. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026–5033 (2012)Google Scholar
  14. 14.
    OpenAI Baselines Implementation.

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Rajesh Mangannavar
    • 1
    Email author
  • Gopalakrishnan Srinivasaraghavan
    • 1
  1. 1.International Institute of Information Technology, BangaloreBangaloreIndia

Personalised recommendations