UCB1 Based Reinforcement Learning Model for Adaptive Energy Management in Buildings

  • Rui Andrade
  • Tiago PintoEmail author
  • Isabel Praça
  • Zita Vale
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 801)


This paper proposes a reinforcement learning model for intelligent energy management in buildings, using a UCB1 based approach. Energy management in buildings has become a critical task in recent years, due to the incentives to the increase of energy efficiency and renewable energy sources penetration. Managing the energy consumption, generation and storage in this domain, becomes, however, an arduous task, due to the large uncertainty of the different resources, adjacent to the dynamic characteristics of this environment. In this scope, reinforcement learning is a promising solution to provide adaptiveness to the energy management methods, by learning with the on-going changes in the environment. The model proposed in this paper aims at supporting decisions on the best actions to take in each moment, regarding buildings energy management. A UCB1 based algorithm is applied, and the results are compared to those of an EXP3 approach and a simple reinforcement learning algorithm. Results show that the proposed approach is able to achieve a higher quality of results, by reaching a higher rate of successful actions identification, when compared to the other considered reference approaches.


Adaptive learning Energy management in buildings EXP3 Reinforcement learning UCB1 


  1. 1.
    Borlase, S.: Smart Grids: Infrastructure, Technology, and Solutions (2012)Google Scholar
  2. 2.
    Rosen, C., Madlener, R.: Regulatory options for local reserve energy markets: implications for prosumers, utilities, and other stakeholders. Energy J. 37, 39–50 (2016)CrossRefGoogle Scholar
  3. 3.
    Fernandes, F., Morais, H., Vale, Z., Ramos, C.: Dynamic load management in a smart home to participate in demand response events. Energy Build. 82, 592–606 (2014)CrossRefGoogle Scholar
  4. 4.
    Acone, M., Romano, R., Piccolo, A., Siano, P., Loia, F., Ippolito, M.G., Zizzo, G.: Designing an energy management system for smart houses. In: 2015 IEEE 15th International Conference on Environment and Electrical Engineering (EEEIC), pp. 1677–1682 (2015)Google Scholar
  5. 5.
    Zhou, B., Li, W., Chan, K.W., Cao, Y., Kuang, Y., Liu, X., Wang, X.: Smart home energy management systems: Concept, configurations, and scheduling strategies. Renew. Sustain. Energy Rev. 61, 30–40 (2016)CrossRefGoogle Scholar
  6. 6.
    Afrakhte, H., Bayat, P., Bayat, P.: Energy management system for smart house with multi-sources using PI-CA controller. In: 2016 Iranian Conference on Renewable Energy Distributed Generation (ICREDG), pp. 24–31 (2016)Google Scholar
  7. 7.
    Pinto, T., Vale, Z., Sousa, T.M., Praça, I., Santos, G., Morais, H.: Adaptive learning in agents behaviour: a framework for electricity markets simulation. Integr. Comput. Aided. Eng. 21, 399–415 (2014)CrossRefGoogle Scholar
  8. 8.
    Li, D., Jayaweera, S.K.: Reinforcement learning aided smart-home decision-making in an interactive smart grid. In: 2014 IEEE Green Energy and Systems Conference (IGESC), pp. 1–6 (2014)Google Scholar
  9. 9.
    Burtini, G., Loeppky, J., Lawrence, R.: A Survey of Online Experiment Design with the Stochastic Multi-Armed Bandit. arXiv1510.00757 [cs, stat] (2015)Google Scholar
  10. 10.
    Bouneffouf, D., Féraud, R.: Multi-armed bandit problem with known trend. Neurocomputing 205, 16–21 (2016)CrossRefGoogle Scholar
  11. 11.
    Shimkin, N.: Reinforcement Learning – Basic Algorithms (2011)Google Scholar
  12. 12.
    Xu, X., Zuo, L., Huang, Z.: Reinforcement learning algorithms with function approximation: recent advances and applications. Inf. Sci. (Ny) 261, 1–31 (2014)MathSciNetCrossRefGoogle Scholar
  13. 13.
    Faia, R., Pinto, T., Abrishambaf, O., Fernandes, F., Vale, Z., Corchado, J.M.: Case based reasoning with expert system and swarm intelligence to determine energy reduction in buildings energy management. Energy Build. 155, 269–281 (2017)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Rui Andrade
    • 1
  • Tiago Pinto
    • 1
    • 2
    Email author
  • Isabel Praça
    • 1
  • Zita Vale
    • 1
  1. 1.GECAD – Research Group, Institute of EngineeringPolytechnic of Porto (ISEP/IPP)PortoPortugal
  2. 2.BISITE Research CentreUniversity of Salamanca (USAL)SalamancaSpain

Personalised recommendations