Reinforcement Learning Testbed for Power-Consumption Optimization

  • Takao MoriyamaEmail author
  • Giovanni De Magistris
  • Michiaki Tatsubori
  • Tu-Hoa Pham
  • Asim Munawar
  • Ryuki Tachibana
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 946)


Common approaches to control a data-center cooling system rely on approximated system/environment models that are built upon the knowledge of mechanical cooling and electrical and thermal management. These models are difficult to design and often lead to suboptimal or unstable performance. In this paper, we show how deep reinforcement learning techniques can be used to control the cooling system of a simulated data center. In contrast to common control algorithms, those based on reinforcement learning techniques can optimize a system’s performance automatically without the need of explicit model knowledge. Instead, only a reward signal needs to be designed. We evaluated the proposed algorithm on the open source simulation platform EnergyPlus. The experimental results indicate that we can achieve 22% improvement compared to a model-based control algorithm built into the EnergyPlus. To encourage the reproduction of our work as well as future research, we have also publicly released an open-source EnergyPlus wrapper interface ( directly compatible with existing reinforcement learning frameworks.


Reinforcement learning Power consumption Data center 


  1. 1.
    Crawley, D.B., et al.: EnergyPlus: creating a new-generation building energy simulation program. Energy Build. 33(4), 319–331 (2001). Special Issue: BUILDING SIMULATION’99. Scholar
  2. 2.
    Dhariwal, P., et al.: OpenAI Baselines (2017).
  3. 3.
    Heess, N., et al.: Emergence of locomotion behaviours in rich environments. arXiv (2017)Google Scholar
  4. 4.
    Inoue, T., Chaudhury, S., De Magistris, G., Dasgupta, S.: Transfer learning from synthetic to real images using variational autoencoders for precise position detection. In: IEEE International Conference on Image Processing (2018)Google Scholar
  5. 5.
    Inoue, T., De Magistris, G., Munawar, A., Yokoya, T., Tachibana, R.: Deep reinforcement learning for high precision assembly tasks. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (2017)Google Scholar
  6. 6.
    Li, Y., Wen, Y., Guan, K., Tao, D.: Transforming cooling optimization for green data center via deep reinforcement learning. arXiv (2017).
  7. 7.
    Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518, 529 (2015)CrossRefGoogle Scholar
  8. 8.
    Munawar, A., et al.: MaestROB: a robotics framework for integrated orchestration of low-level control and high-level reasoning. In: IEEE International Conference on Robotics and Automation. IEEE (2018)Google Scholar
  9. 9.
    Pham, T.H., De Magistris, G., Tachibana, R.: OptLayer - practical constrained optimization for deep reinforcement learning in the real world. In: IEEE International Conference on Robotics and Automation (2018)Google Scholar
  10. 10.
  11. 11.
    Schulman, J., Levine, S., Abbeel, P., Jordan, M., Moritz, P.: Trust region policy optimization. In: Proceedings of International Conference on Machine Learning (2015)Google Scholar
  12. 12.
    Sun, J., Reddy, A.: Optimal control of building HVAC&R systems using complete simulation-based sequential quadratic programming (CSB-SQP). Build. Environ. 40(5), 657–669 (2005). Scholar
  13. 13.
    Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., Abbeel, P.: Domain randomization for transferring deep neural networks from simulation to the real world. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (2017)Google Scholar
  14. 14.
    Wei, T., Wang, Y., Zhu, Q.: Deep reinforcement learning for building HVAC control. In: 2017 54th ACM/EDAC/IEEE Design Automation Conference (DAC), pp. 1–6, June 2017.

Copyright information

© Springer Nature Singapore Pte Ltd. 2018

Authors and Affiliations

  • Takao Moriyama
    • 1
    Email author
  • Giovanni De Magistris
    • 1
  • Michiaki Tatsubori
    • 1
  • Tu-Hoa Pham
    • 1
  • Asim Munawar
    • 1
  • Ryuki Tachibana
    • 1
  1. 1.IBM Research – TokyoTokyoJapan

Personalised recommendations