Smart Cities pp 177-205 | Cite as

Big Energy Data Management for Smart Grids—Issues, Challenges and Recent Developments

  • Vidyasagar Potdar
  • Anulipt Chandan
  • Saima Batool
  • Naimesh Patel
Part of the Computer Communications and Networks book series (CCN)


Urban areas suffer from tremendous pressure to cope with increasing population in a city. A smart city is a technological solution that integrates engineering and information systems to assist in managing these scarce resources. A smart city comprises several intelligent services such as smart grids, smart education, smart transportation, smart buildings, smart waste management and so on. Among all these, smart grids are the nucleus of all the facilities because these provide sustainable electrical supply for other smart services to operate seamlessly. Smart grids integrate information and communication technologies (ICT) into traditional energy grids, thereby capturing massive amounts of data from several devices like smart meters, sensors, and other electrical infrastructures. The data collected in smart grids are heterogeneous and require data analytic techniques to extract meaningful information to make informed decisions. We term this enormous amount of data as big energy data. This book chapter discusses progress in the field of big energy data by enlisting different studies that cover several data management aspects such as data collection, data preprocessing, data integration, data storage, data analytics, data visualisation and decision-making. We also discuss various challenges in data management and report recent progress in this field. Finally, we present open research areas in big data management especially in relation to smart grids.


Smart city Smart grid Big energy data Data management Smart meter Energy data management Data lifecycle Data preprocessing Data collection Data integration Data storage Data analytics Data visualisation Decision-making 


  1. 1.
    Gantz J, Reinsel D (2011) Extracting value from chaos. IDC IView 1–12Google Scholar
  2. 2.
    Tushar W, Zhang JA, Smith DB et al (2014) Prioritizing consumers in smart grid: a game theoretic approach. IEEE Trans Smart Grid 5:1429–1438CrossRefGoogle Scholar
  3. 3.
    Zhou K, Fu C, Yang S (2016) Big data driven smart energy management: from big data to big insights. Renew Sustain Energy Rev 56:215–225. Scholar
  4. 4.
    Chen M, Mao S, Liu Y (2014) Big data: a survey. Mob Netw Appl 19:171–209. Scholar
  5. 5.
    Knight S, Burn J (2005) Developing a framework for assessing information quality on the world wide web. Informing Sci 8:159–172CrossRefGoogle Scholar
  6. 6.
    Aman S, Chelmis C, Prasanna V (2014) Addressing data veracity in big data applications. IEEE, pp 1–3Google Scholar
  7. 7.
    Uludag S, Lui K-S, Ren W, Nahrstedt K (2016) Secure and scalable data collection with time minimization in the smart grid. IEEE Trans Smart Grid 7:43–54CrossRefGoogle Scholar
  8. 8.
    Australian Bureau of Statistics Dwelling Type (2017) Greater Sydney. In: Id Popul. Experts. Accessed 1 Nov 2017
  9. 9.
    Daki H, El Hannani A, Aqqal A et al (2017) Big Data management in smart grid: concepts, requirements and implementation. J Big Data 4:13. Scholar
  10. 10.
    Fehrenbacher K (2012) 10 ways big data is remaking energy. Accessed 1 Nov 2017
  11. 11.
    Li X, Lv Z, Hu J et al (2015) XEarth: a 3D GIS Platform for managing massive city information. IEEE, pp 1–6Google Scholar
  12. 12.
    Farhangi H (2010) The path of the smart grid. IEEE Power Energy Mag 8Google Scholar
  13. 13.
    Domínguez J, Amador J (2007) Geographical information systems applied in the field of renewable energy sources. Comput Ind Eng 52:322–326CrossRefGoogle Scholar
  14. 14.
    Sánchez-Lozano JM, Teruel-Solano J, Soto-Elvira PL, García-Cascales MS (2013) Geographical information systems (GIS) and multi-criteria decision making (MCDM) methods for the evaluation of solar farms locations: case study in south-eastern Spain. Renew Sustain Energy Rev 24:544–556CrossRefGoogle Scholar
  15. 15.
    Amador J, Domínguez J (2005) Application of geographical information systems to rural electrification with renewable energy sources. Renew Energy 30:1897–1912CrossRefGoogle Scholar
  16. 16.
    Cody R (2017) Cody’s data cleaning techniques using SAS. SAS InstituteGoogle Scholar
  17. 17.
    Chu X, Ilyas IF (2016) Qualitative data cleaning. Proc VLDB Endowment 9:1605–1608CrossRefGoogle Scholar
  18. 18.
    Sagiroglu S, Terzi R, Canbay Y, Colak I (2016) Big data issues in smart grid systems. IEEE, pp 1007–1012Google Scholar
  19. 19.
    Zhang N, Yang X, Zhang M, Long K (2016) RMI-DRE: a redundancy-maximizing identification scheme for data redundancy elimination. Sci China Inf Sci 59:089301CrossRefGoogle Scholar
  20. 20.
    Xie S, Chen Z (2017) Anomaly detection and redundancy elimination of big sensor data in internet of things. ArXiv Prepr ArXiv170303225Google Scholar
  21. 21.
    Lee K, Kim D, Shin I (2017) Improving throughput in wireless networks using redundancy elimination. IEEE Commun Lett 21:160–163CrossRefGoogle Scholar
  22. 22.
    Chen W, Zhou K, Yang S, Wu C (2017) Data quality of electricity consumption data in a smart grid environment. Renew Sustain Energy Rev 75:98–105CrossRefGoogle Scholar
  23. 23.
    Guerrero JI, García A, Personal E et al (2017) Heterogeneous data source integration for smart grid ecosystems based on metadata mining. Expert Syst Appl 79:254–268CrossRefGoogle Scholar
  24. 24.
    Taieb SB, Huser R, Hyndman RJ, Genton MG (2016) Forecasting uncertainty in electricity smart meter data by boosting additive quantile regression. IEEE Trans Smart Grid 7:2448–2455CrossRefGoogle Scholar
  25. 25.
    Jurado S, Nebot À, Mugica F, Mihaylov M (2017) Fuzzy inductive reasoning forecasting strategies able to cope with missing data: a smart grid application. Appl Soft Comput 51:225–238CrossRefGoogle Scholar
  26. 26.
    Alahakoon D, Yu X (2016) Smart electricity meter data intelligence for future energy systems: a survey. IEEE Trans Ind Inform 12:425–436CrossRefGoogle Scholar
  27. 27.
    Neumann S, Wilhoit F, Goodrich M, Balijepalli VM (2016) Everything’s talking to each other: smart meters generate big data for utilities and customers. IEEE Power Energy Mag 14:40–47CrossRefGoogle Scholar
  28. 28.
    Ahsan U, Bais A (2017) Distributed big data management in smart grid. IEEE, pp 1–6Google Scholar
  29. 29.
    Qiu RC, Antonik P (2014) Big data for smart grid. Smart Grid Using Big Data Anal Random Matrix Theory Approach 485–491Google Scholar
  30. 30.
    Mosaddegh A, Cañizares CA, Bhattacharya K, Fan H (2017) Distributed computing architecture for optimal control of distribution feeders with smart loads. IEEE Trans Smart Grid 8:1469–1478CrossRefGoogle Scholar
  31. 31.
    Cao Z, Lin J, Wan C et al (2017) Hadoop-based framework for big data analysis of synchronised harmonics in active distribution network. IET Gener Transm Distrib 11(16):3930–3937Google Scholar
  32. 32.
    Park C-K, Kim H-J, Kim Y-S (2014) A study of factors enhancing smart grid consumer engagement. Energy Policy 72:211–218CrossRefGoogle Scholar
  33. 33.
    Eltantawy AB, Salama MM (2014) A novel zooming algorithm for distribution load flow analysis for smart grid. IEEE Trans Smart Grid 5:1704–1711CrossRefGoogle Scholar
  34. 34.
    Weng Y, Negi R, Faloutsos C, Ilić MD (2017) Robust data-driven state estimation for smart grid. IEEE Trans Smart Grid 8:1956–1967CrossRefGoogle Scholar
  35. 35.
    Moradi MH, Eskandari M, Hosseinian SM (2015) Operational strategy optimization in an optimal sized smart microgrid. IEEE Trans Smart Grid 6:1087–1095CrossRefGoogle Scholar
  36. 36.
    Jiang H, Zhang JJ, Gao W, Wu Z (2014) Fault detection, identification, and location in smart grid based on data-driven computational methods. IEEE Trans Smart Grid 5:2947–2956CrossRefGoogle Scholar
  37. 37.
    Diamantoulakis PD, Kapinas VM, Karagiannidis GK (2015) Big data analytics for dynamic energy management in smart grids. Big Data Res 2:94–101CrossRefGoogle Scholar
  38. 38.
    Srinivasan D, Reindl T (2015) Real-time display of data from a smart-grid on geographical map using a GIS tool and its role in optimization of game theory. IEEE, pp 1–6Google Scholar
  39. 39.
    Stefan M, Lopez JG, Andreasen MH, Olsen RL (2017) Visualization techniques for electrical grid smart metering data: a survey. In: Proceedings of the 2017 IEEE third international conference on big data. Computing service and applications (BigDataService)Google Scholar
  40. 40.
    Yu M, Hong SH (2016) A real-time demand-response algorithm for smart grids: a stackelberg game approach. IEEE Trans Smart Grid 7:879–888Google Scholar
  41. 41.
    Siano P (2014) Demand response and smart grids—a survey. Renew Sustain Energy Rev 30:461–478CrossRefGoogle Scholar
  42. 42.
    Colson CM, Nehrir MH (2013) Comprehensive real-time microgrid power management and control with distributed agents. IEEE Trans Smart Grid 4:617–627CrossRefGoogle Scholar
  43. 43.
    Akhavan-Rezai E, Shaaban MF, El-Saadany EF, Karray F (2016) Online intelligent demand management of plug-in electric vehicles in future smart parking lots. IEEE Syst J 10:483–494CrossRefGoogle Scholar
  44. 44.
    Liu N, Chen J, Zhu L et al (2013) A key management scheme for secure communications of advanced metering infrastructure in smart grid. IEEE Trans Ind Electron 60:4746–4756CrossRefGoogle Scholar
  45. 45.
    Uludag S, Zeadally S, Badra M (2015) Techniques, taxonomy, and challenges of privacy protection in the smart grid. In: Privacy in a digital, networked world. Springer, Berlin, pp 343–390Google Scholar
  46. 46.
    Fouda MM, Fadlullah ZM, Kato N et al (2011) A lightweight message authentication scheme for smart grid communications. IEEE Trans Smart Grid 2:675–685CrossRefGoogle Scholar
  47. 47.
    Bekara C, Luckenbach T, Bekara K (2012) A privacy preserving and secure authentication protocol for the advanced metering infrastructure with non-repudiation service. Proc ENERGYGoogle Scholar
  48. 48.
    Abbasinezhad-Mood D, Nikooghadam M (2017) An ultra-lightweight and secure scheme for communications of smart meters and neighborhood gateways by utilization of an ARM Cortex-M Microcontroller. IEEE Trans Smart GridGoogle Scholar
  49. 49.
    Kim Y-J, Kolesnikov V, Kim H, Thottan M (2011) SSTP: a scalable and secure transport protocol for smart grid data collection. IEEE, pp 161–166Google Scholar
  50. 50.
    Yukun N, Xiaobin T, Shi C et al (2013) A security privacy protection scheme for data collection of smart meters based on homomorphic encryption. IEEE, pp 1401–1405Google Scholar
  51. 51.
    Li F, Luo B, Liu P (2010) Secure information aggregation for smart grids using homomorphic encryption. IEEE, pp 327–332Google Scholar
  52. 52.
    Saputro N, Akkaya K (2017) Investigation of smart meter data reporting strategies for optimized performance in smart grid AMI networks. IEEE Internet Things J 4:894–904CrossRefGoogle Scholar
  53. 53.
    Saranya P, Ponmagal R (2016) Secure and scalable sensor data access network with delay tolerance. Wireless Commun 8:202–206Google Scholar
  54. 54.
    Wang T, Zeng J, Lai Y et al (2017) Data collection from WSNs to the cloud based on mobile Fog elements. Future Gener Comput SystGoogle Scholar
  55. 55.
    Aiello M, Pagani GA (2014) The smart grid’s data generating potentials. IEEE, pp 9–16Google Scholar
  56. 56.
    Uddin F (2017) Energy-aware optimal data aggregation in smart grid wireless communication networks. IEEE Trans Green Commun Netw 1:358–371CrossRefGoogle Scholar
  57. 57.
    Groat MM, Hey W, Forrest S (2011) KIPDA: k-indistinguishable privacy-preserving data aggregation in wireless sensor networks. IEEE, pp 2024–2032Google Scholar
  58. 58.
    Rottondi C, Verticale G, Krauss C (2013) Distributed privacy-preserving aggregation of metering data in smart grids. IEEE J Sel Areas Commun 31:1342–1354CrossRefGoogle Scholar
  59. 59.
    Li F, Luo B (2012) Preserving data integrity for smart grid data aggregation. IEEE, pp 366–371Google Scholar
  60. 60.
    Jin H, Uludag S, Lui K-S, Nahrstedt K (2014) Secure data collection in constrained tree-based smart grid environments. IEEE, pp 308–313Google Scholar
  61. 61.
    Karimi B, Namboodiri V, Jadliwala M (2015) Scalable meter data collection in smart grids through message concatenation. IEEE Trans Smart Grid 6:1697–1706CrossRefGoogle Scholar
  62. 62.
    Bartoli A, Hernandez-Serrano J, Soriano M et al (2010) Secure lossless aggregation for smart grid M2M networks. IEEE, pp 333–338Google Scholar
  63. 63.
    Niyato D, Wang P (2012) Cooperative transmission for meter data collection in smart grid. IEEE Commun Mag 50Google Scholar
  64. 64.
    Shao S, Guo S, Qiu X et al (2014) Traffic scheduling for wireless meter data collection in smart grid communication network. IEEE, pp 1–7Google Scholar
  65. 65.
    Dhasian HR, Balasubramanian P (2013) Survey of data aggregation techniques using soft computing in wireless sensor networks. IET Inf Secur 7:336–342CrossRefGoogle Scholar
  66. 66.
    Yan Y, Qian Y, Sharif H (2011) A secure data aggregation and dispatch scheme for home area networks in smart grid. IEEE, pp 1–6Google Scholar
  67. 67.
    Tavasoli M, Yaghmaee MH, Mohajerzadeh AH (2016) Optimal placement of data aggregators in smart grid on hybrid wireless and wired communication. IEEE, pp 332–336Google Scholar
  68. 68.
    Pinheiro G, Vinagre E, Praça I et al (2018) Smart grids data management: a case for cassandra. In: 14th international conference on distributed computing and artificial intelligence. Springer, Berlin, p 87Google Scholar
  69. 69.
    De Virgilio R (2017) Smart RDF data storage in graph databases. In: Proceedings of the 17th IEEE/ACM international symposium on cluster, cloud and grid computing. IEEE Press, pp 872–881Google Scholar
  70. 70.
    Yang H, Li P, Masood A et al (2016) Smart grid data analysis and prediction modeling. In: 2016 international conference on progress in informatics and computing (PIC). IEEE, pp 541–544Google Scholar
  71. 71.
    Arora S, Taylor JW (2016) Forecasting electricity smart meter data using conditional kernel density estimation. Omega 59:47–59CrossRefGoogle Scholar
  72. 72.
    Zhou Y, Li P, Xiao Y et al (2016) Smart grid data mining and visualization. In: 2016 international conference on progress in informatics and computing (PIC). IEEE, pp 536–540Google Scholar
  73. 73.
    Calvillo CF, Sánchez-Miralles A, Villar J (2016) Energy management and planning in smart cities. Renew Sustain Energy Rev 55:273–287. Scholar
  74. 74.
    Ruiz-Romero S, Colmenar-Santos A, Mur-Pérez F, López-Rey Á (2014) Integration of distributed generation in the power distribution network: the need for smart grid control systems, communication and equipment for a smart city—use cases. Renew Sustain Energy Rev 38:223–234. Scholar
  75. 75.
    Haidar AMA, Muttaqi K, Sutanto D (2015) Smart grid and its future perspectives in Australia. Renew Sustain Energy Rev 51:1375–1389. Scholar
  76. 76.
    Fadaeenejad M, Saberian AM, Fadaee M et al (2014) The present and future of smart power grid in developing countries. Renew Sustain Energy Rev 29:828–834. Scholar
  77. 77.
    Welsch M, Bazilian M, Howells M et al (2013) Smart and just grids for sub-Saharan Africa: exploring options. Renew Sustain Energy Rev 20:336–352. Scholar
  78. 78.
    Lee AH, Chen HH, Chen J (2017) Building smart grid to power the next century in Taiwan. Renew Sustain Energy Rev 68:126–135CrossRefGoogle Scholar
  79. 79.
    IqtiyaniIlham N, Hasanuzzaman M, Hosenuzzaman M (2017) European smart grid prospects, policies, and challenges. Renew Sustain Energy Rev 67:776–790CrossRefGoogle Scholar
  80. 80.
    Di Santo KG, Kanashiro E, Di Santo SG, Saidel MA (2015) A review on smart grids and experiences in Brazil. Renew Sustain Energy Rev 52:1072–1082CrossRefGoogle Scholar
  81. 81.
    Ponce P, Polasko K, Molina A (2016) End user perceptions toward smart grid technology: acceptance, adoption, risks, and trust. Renew Sustain Energy Rev 60:587–598CrossRefGoogle Scholar
  82. 82.
    Sun Y-Y, Yuan J-J, Zhai M-Y (2016) Cloud-based data analysis of user side in smart grid. IEEE, pp 39–44Google Scholar
  83. 83.
    Chou J-S, Ngo N-T (2016) Smart grid data analytics framework for increasing energy savings in residential buildings. Autom Constr 72:247–257CrossRefGoogle Scholar
  84. 84.
    Haider HT, See OH, Elmenreich W (2016) A review of residential demand response of smart grid. Renew Sustain Energy Rev 59:166–178. Scholar
  85. 85.
    Wang W, Lu Z (2013) Cyber security in the smart grid: survey and challenges. Comput Netw 57:1344–1371CrossRefGoogle Scholar
  86. 86.
    Tan S, De D, Song W-Z et al (2017) Survey of security advances in smart grid: a data driven approach. IEEE Commun Surv Tutor 19:397–422CrossRefGoogle Scholar
  87. 87.
    Kosut O, Jia L, Thomas RJ, Tong L (2010) Malicious data attacks on smart grid state estimation: attack strategies and countermeasures. IEEE, pp 220–225Google Scholar
  88. 88.
    Xie L, Mo Y, Sinopoli B (2011) Integrity data attacks in power market operations. IEEE Trans Smart Grid 2:659–666CrossRefGoogle Scholar
  89. 89.
    Xie L, Mo Y, Sinopoli B (2010) False data injection attacks in electricity markets. IEEE, pp 226–231Google Scholar
  90. 90.
    Sooriyabandara M, Ekanayake J (2010, December) Smart grid-technologies for its realisation. In: IEEE international conference on sustainable energy technologies (ICSET) 2010. IEEE, pp 1–4Google Scholar
  91. 91.
    Wang Y, Li W, Lu J (2010) Reliability analysis of wide-area measurement system. IEEE Trans Power Deliv 25:1483–1491CrossRefGoogle Scholar
  92. 92.
    Moslehi K, Kumar R (2010) A reliability perspective of the smart grid. IEEE Trans Smart Grid 1:57–64CrossRefGoogle Scholar
  93. 93.
    Yan Y, Qian Y, Sharif H, Tipper D (2013) A survey on smart grid communication infrastructures: motivations, requirements and challenges. IEEE Commun Surv Tutor 15:5–20CrossRefGoogle Scholar
  94. 94.
    Tonyali S, Akkaya K, Saputro N et al (2018) Privacy-preserving protocols for secure and reliable data aggregation in IoT-enabled smart metering systems. Future Gener Comput Syst 78:547–557CrossRefGoogle Scholar
  95. 95.
    Card SK, Mackinlay JD, Shneiderman B (1999) Readings in information visualization: using vision to think. Morgan Kaufmann Publishers, San FranciscoGoogle Scholar
  96. 96.
    Ware C (2012) Information visualization: perception for design. ElsevierCrossRefGoogle Scholar
  97. 97.
    Zhu J, Zhuang E, Fu J et al (2016) A framework-based approach to utility big data analytics. IEEE Trans Power Syst 31:2455–2462CrossRefGoogle Scholar
  98. 98.
    Liu H, Guo J, Yu W et al (2016) The design and implementation of the enterprise level data platform and big data driven applications and analytics. IEEE, pp 1–5Google Scholar
  99. 99.
    Ma K-L, Stompel A, Bielak J et al (2003) Visualizing very large-scale earthquake simulations. In: 2003 ACM/IEEE conference on supercomputing. IEEE, pp 48–48Google Scholar
  100. 100.
    Lamping J, Rao R, Pirolli P (1995) A focus + context technique based on hyperbolic geometry for visualizing large hierarchies. In: Proceedings of the SIGCHI conference on human factors in computing systems. ACM Press/Addison-Wesley Publishing Co., pp 401–408Google Scholar
  101. 101.
    Johnson B, Shneiderman B (1991) Tree-maps: a space-filling approach to the visualization of hierarchical information structures. In: Proceedings of the 2nd conference on Visualization ’91. IEEE Computer Society Press, pp 284–291Google Scholar
  102. 102.
    Tam NT, Song I (2016) Big data visualization. In: Information science and applications (ICISA) 2016. Springer, pp 399–408Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Vidyasagar Potdar
    • 1
  • Anulipt Chandan
    • 2
  • Saima Batool
    • 1
  • Naimesh Patel
    • 3
  1. 1.Curtin Business SchoolCurtin UniversityPerthAustralia
  2. 2.National Institute of TechnologyAgartalaIndia
  3. 3.Safeworld Systems Pvt LtdAhmedabadIndia

Personalised recommendations