Electricity demand or load forecasts inform both industrial and governmental decision making processes, from energy trading and electricity pricing to demand response and infrastructure maintenance. Electric load forecasts allow Distribution System Operators (DSOs) and policy makers to prepare for the short and long term future. For informed decisions to be made, particularly within industries that are highly regulated such as electricity trading, the factors influencing electricity demand need to be understood well. This is only becoming more urgent, as low carbon technologies (LCT) become more prevalent and consumers start to generate electricity for themselves, trade with peers and interact with DSOs.

In order to understand and meet demands effectively, smart grids are being developed in many countries, including the UK, collecting high resolution data and making it more readily accessible. This data allows for better analysis of demand, identification of issues and control of electric networks. Indeed, using high quality forecasts is one of the most common ways to understand demand and with smart meter data, it is possible to know not only how much electricity is required at very high time resolutions, but also how much is required at the substation, feeder and/or household level.

However, this poses new challenges. Mainly, load profiles of individual households and substations are harder to predict than aggregated regional or national load profiles due to their volatile nature. The load profiles at the low voltage level contain irregular peaks, or local maxima, which are smoothed out when averaged across space or time. Once aggregated, the profiles are smoother, and easier to forecast. The work presented in this book outlines the challenge of forecasting energy demand at the individual level and aims to deepen our understanding of how better to forecast peaks which occur irregularly.

Even more uncertainty arises from the drastic changes to the way society has used electricity thus far and will do so in the future. Many communities have moved away from gas and coal powered technologies to electrically sourced ones, especially for domestic heating [1]. Moreover, where households and businesses were previously likely to be consumers, government policies incentivising solar energy have lead to an increase in photovoltaic panel installations [2], meaning that the interaction with the electricity grid/ DSO will become increasingly dynamic.

In addition to this, governments are also diversifying the source of electricity generation, i.e. with renewable and non-renewable sources [3, 4] and incentivising the purchase of electric vehicles [5] in a bid to reduce national and global greenhouse emissions. This evolution of societal behaviour as well as governmental and corporate commitments to combat climate change is likely to add more volatility to consumption patterns [6] and thereby increase uncertainty. Most likely the changing climate itself will drive different human behaviours to current ones and introduce yet more unknowns to the problem. Therefore, while the literature on forecasting of electricity load is large and growing, there is a definite need to revisit the topic to address these issues. As demand response, battery control and peer-to-peer energy trading are all very sensitive to peaks at the individual or residential level, particular attention will be given to forecasting the peaks in low-voltage load profiles.

While the change of attention from average load to peak load is not new, a novel approach in terms of electricity load forecasting, is to adapt the techniques from a branch of statistics known as Extreme Value Theory (EVT). We will speak in depth about it in later chapters but we briefly share a sense of its scope and our vision for its application to the electricity demand forecasting literature. We can use the methods from EVT to study the bad-case and the worst-case scenarios, such as blackouts which, though rare, are inevitable and highly disruptive. Not just households [7] but businesses [8] and even governments [9] may be vulnerable to risks from blackouts or power failure. In order to increase resilience and guard against such high impact events, businesses in particular may consider investing in generators or electricity storage devices. However, these technologies are currently expensive and the purchase of these may need to be justified through rigorous cost-benefit analyses. We believe that the techniques presented in this book and that to be developed throughout the course of this project could be used by energy consultants to assess such risks and to determine optimal electricity packages for businesses and individuals.

As one of our primary goals is to study extremes in electricity load profiles and incorporate this into forecasts for better accuracy, we will first consider the forecasting algorithms that are commonly suggested in the literature and how and where these algorithms fail. The latter will be done by (1) considering different error measures (the classic approach in load forecasting) and (2) by studying “heteroscedasticity” in forecasts errors (an EVT approach), which for the moment can be understood as the irregular frequency of large errors or even the inability of the algorithm to predict accurately over time. We will also estimate the upper bound of the demand. We believe that DSOs will be able to use these kinds of techniques to realistically assess what contractual obligations to place upon individual customers and thereby tailor their contracts. They may also prove useful in demand response strategies.

In this book, we will consider two smart meter data sets; the first is from smart meter trials in Ireland and the second is collected as part of the Thames Valley Vision (TVV) Project in the UK. The Irish smart meter trials is available publicly and so has been used in many journal papers and is a good starting point. However, little information about the households is available. The TVV Project on the other hand is geographically compressed on a relatively small area, allowing weather and other data about the area to be collected. The substation data is available at higher time resolution than the Irish smart meter data and subsequently provides more information with which to build statistical models. Combining both the classic forecasts with the results from EVT, we aim to set benchmarks and describe the extreme behaviour.

While both case studies relate to energy, particularly electricity, the methods presented here are by no means exclusively for this sector; they can be and have been applied more broadly as we will see in later chapters. Thus, the work presented in this book may also serve to illustrate how results from EVT can be adapted to different disciplines. Furthermore, this book may also prove conducive to learning how to visualise and understand large amounts data and checking of underlying assumptions. In order to facilitate adaptations to other applications and generally share knowledge, some of the code used in this work has been made accessible through GitHubFootnote 1 so those teaching or attending data science courses may use it to create exercises extending the code, or to run experiments on different data-sets.

1.1 Forecasting and Challenges

Electricity load forecasts can be generated for minutes and hours in advance to years and decades in advance. Forecasts of different lengths assist in different applications, for example forecasts for up to a day ahead are generated for the purpose of demand response or battery control, whereas daily to yearly forecasts may be produced for energy trading, and yearly to decade forecasts allow for grid maintenance and investment planning and informing energy policy (Fig. 1.1).

Fig. 1.1
figure 1

The various classifications for electric load forecasts and their applications. Based on: Hong and Fan [10]. The abbreviations are Short Term Load Forecasting (STLF), and Long Term Load Forecasting (LTLF)

Full size image

Most studies in electric load forecasting in the past century have focused on point load forecasting, meaning that at each time point, one value is provided, usually an average. The decision making process in the utility industry relies mostly on expected values (averages) so it is no surprise that these types of forecasts have been the dominant tool in the past. However, market competition and requirements to integrate renewable technology have inspired interest in probabilisitic load forecasts (PLF) particularly for system planning and operations. PLF may use quantiles, intervals and/or density functions [10]. We will review the forecast literature in more detail in Chap. 2, focusing mostly on point/deterministic forecasts. It is worth noting that many of those point-forecast methods can be implemented for quantiles prediction.

It becomes evident from various electric load forecasting reviews presented by Gerwig [11], Alfares and Nazeeruddin [12], Hong and Fan [10], that many algorithms of varying complexity exist in the literature. However, for many reasons they are not always particularly good in predicting peaks [13]. The fundamental idea behind most forecasting algorithms is that a future day (or time) is likely to be very much like days (or times) in the past that were similar to it with regard to weather, season, day of the week, etc. Thus, algorithms mostly use averaging or regression techniques to generate forecasts. This brings us back to the first challenge mentioned earlier: such algorithms work well when the demand profiles are smooth, for example due to aggregation at the regional and/or national level, but when the profiles are irregular and volatile, the accuracy of forecasts is reduced. This is usually the case for households or small feeder (sometimes called residential) profiles. In this way, it becomes obvious that we need algorithms that can recreate peaks in the forecasts that are representative of the peaks in the observed profiles.

This brings us to the second challenge: in order to determine which algorithms perform well and which perform better (or worse), we need to establish benchmarks and specify how we measure accuracy. There are many ways of assessing the quality of forecasts, or more strictly many error metrics that may be used. Some conventional error metrics for load forecasts are mean absolute percentage error (MAPE) and mean absolute error (MAE) (see Sect. 2.2.1). These are reasonably simple and transparent and thus quite favourable in the electric load forecasting community. However, as noted by Haben et al. [14], for low-voltage networks, a peaky forecast is more desirable and realistic than a flat one but error metrics such as MAPE unjustly penalise peaky forecasts and can often quantify a flat forecast to be better. This is because the peaky forecast is penalised twice: once for missing the observed peak and again for forecasting it to be where it did not occur, even if only slightly shifted in time. Thus, some other error measures have been devised recently that tackle this issue. We will review these more in Chap. 2.

Both of these challenges can also be approached from an EVT point of view. On the one hand, peaks in the data can be thought of as local extremes. By considering how large the observations can feasibly become in future, we may be able to quantify how likely it is that observations exceed some large threshold. Equally, as discussed before, we can use heteroscedasticity to describe how behaviour deviates from the “typical” in time, which may help us to understand if particular time windows are hard to predict, thereby assessing uncertainty.

Ultimately, we want to combine the knowledge from both these branches and improve electricity forecasts for each household. Of course, improving forecasts of individual households will improve forecasting ability overall, but DSOs are also interested in understanding how demand evolves in time and the limits of consumption. How much is a customer ever likely to use? When are peaks likely to happen? How long will they last? Knowing this at the household level can help DSOs to incentivise flexibility, load spreading or ‘peak shaving’. Such initiatives encourage customers to use less electricity when it is in high demand. Load spreading informed only by regional and national load patterns may prove counter productive at the substation level; for example, exclusive night time charging of electric vehicles, as this is when consumption is nationally low, without smart algorithms or natural diversity may make the substations or feeders vulnerable to night time surges, as pointed out in Hattam et al. [15]. Thus, understanding local behaviour is important to both informing policy and providing personalised customer services.

Before we delve into the theory and methods, we familiarise ourselves with Irish smart meter data in Sect. 1.2.1 and with the TVV data in Sect. 1.2.2.

1.2 Data

1.2.1 Irish Smart Meter Data

The first case study uses data obtained from Irish Social Science Data Archive [16]. The Smart Metering Project was launched in Ireland in 2007 with the intention of understanding consumer behaviour with regard to the influence of smart meter technology. To aid this investigation, smart meters were installed in roughly 5000 households. Trials with different interventions were ran for groups of households. The data used in this book are from those households, which were used as controls in the trials. Therefore, they were not taking part in any intervention (above and beyond a smart meter installation). This gives complete measurements for 503 households. We have further subset the data to use only 7 weeks, starting in August 2010, where the weeks are labelled from 16 to 22 (inclusive). No bank holidays or other national holidays were observed in this period. Measurements were taken at half hourly resolution which are labelled from 1 to 48 where 1 is understood to correspond to midnight. Additionally days are also numbered from 593 (16th of August 2010) to 641. From this, the days of the weeks, ranging from 1 to 7 where 1 is Monday and 7 is Sunday, were deduced. Regardless of the number of occupants, each household is considered to be the unit and the terminology of “customer” and “household” are used interchangeably and equivalently throughout.

We now familiarise ourselves with the data at hand. Consider both the histogram and the box plot shown in Fig. 1.2. The 75th percentile for this data is 0.5 kWh meaning that three quarters of the observations are below this value, however some measurements are as high as 12 kWh. Generally, large load values can be attributed to consumers operating a small business from home, having electric heating, multiple large appliances and/or electric vehicles in their home. However, electric vehicle recharging does not seem to be a plausible explanation in this data set as it is a recurring, constant and prolonged activity and such a sustained demand was not observed in any of the profiles. Other large values are roughly between 9 and 10 kWh so we may ask ourselves, what caused such a large surge? Was it a one time thing? How large can that value get within reason? How long can it last? We will address this specific question when we consider “endpoint estimation” in Chap. 4 and for which the theoretical background will be reviewed in Chap. 3.

Fig. 1.2
figure 2

Histogram, logarithmic y scale, and box-plot of half hourly measurements in Irish smart meter data

Full size image
Fig. 1.3
figure 3

Cumulative demand profiles in kiloWatt hours (kWh) for various time horizons

Full size image

While Fig. 1.2 tells us about half hourly demand, Fig. 1.3 gives some general profiles. These four plots show the total/cumulative pattern of electricity demand. The top left plot in Fig. 1.3 shows the dip in usage overnight, the increase for breakfast which stabilises during typical working hours with a peak around lunch and rises finally again for dinner, which is when it is at its highest on average. Similarly, the top right plot of Fig. 1.3 shows the total daily consumption for each day in the 7 week period. The plot highlights a recurring pattern which indicates that there are specific days in the week where usage is relatively high and others where it is low. This is further confirmed by the image on the bottom left which tells us that, in total, Fridays tend to have the lowest load, whereas weekends typically have the highest. Finally, the image on the bottom right shows a rise in demand starting in week 18, which is around the beginning of September, aligning with the start of the academic year for all primary and some secondary schools in Ireland. This explains why the jump in data occurs as the weeks preceding are weeks when many families may travel abroad and thus record less electricity demand in their homes.

Fig. 1.4
figure 4

Total load profiles for each day of the week

Full size image

It is also valuable to see how the top left profile of Fig. 1.3 changes for each day of the week. From Fig. 1.4, it is obvious that there are some differences between weekdays and weekends; the breakfast peak is delayed on weekends but no categorical differences are obvious for the evening peaks between weekends and weekdays. Notice that both the top left image of Fig. 1.3 and the weekday profiles in Fig. 1.4 show three peaks: one for breakfast around 8 am, another for lunch around 1 pm and the third in the evening which is sustained for longer. While we are not currently exploring the impact and benefits of clustering, we may use these three identifiers to cluster households by their usage in the future.

Already, we can see the basis for the most forecasting algorithms that we mentioned before. When profiles are averaged, they are smooth and thus overall averaging techniques may work well. Furthermore, if most Sundays record high usage, then it is sensible to use profiles from past Sundays to predict the demand for future Sundays, i.e. to use similar days.

In a similar way, it may be sensible to use similar time windows on corresponding days, that is using past Sunday evenings to predict future Sunday evenings. One way to see if this holds in practice as well as in principle is to consider correlation. Figure 1.5 shows the relationship between the daily demand of each household on day d against the daily demand on day \(d-1\). Each marker indicates a different household though it should be noted that there is not a unique colour for each. There seems to be evidence of a somewhat linear trend and some variation which may be resulting from the fact that weekends have not been segregated from weekdays and we are not always comparing similar days. To see how far back this relationship holds, an auto-correlation function (Fig. 1.6) is provided. The auto-correlation function is for the aggregated series given by the arithmetic mean of all customers, \(\frac{1}{n} \sum _{i=1}^{n} x_i\), where \(x_i\) is the load of the ith household, at each half hour. The dashed line represents the 95% confidence interval. As can be seen, there is some symmetry and while it is not shown here there is also periodicity throughout the data set though with decreasing auto-correlation. This gives us the empirical foundation to use many of the forecasts which rely on periodicity for accuracy.

Fig. 1.5
figure 5

Electric load day d against day \(d-1\) in kWh

Full size image
Fig. 1.6
figure 6

Auto-correlation function for 1 day. Lag is measured in half hour

Full size image

Finally, and as a prelude to what follows in Chap. 5, one way to see if there are “extreme” households is to consider the daily total demand of each household. This is shown in Fig. 1.7, again with each marker representing different households as before. It is noteworthy that there is one house (coloured in light blue) that consistently appears to be using the most amount of electricity per day. This may be an example of a household where the occupants are operating a small business from home.

Fig. 1.7
figure 7

Total daily demand for each household

Full size image

1.2.2 Thames Valley Vision Data

This second case study uses data that was collected as a part of Scottish and Southern Electricity Network (SSEN) Thames Valley Vision project (TVV),Footnote 2 funded by the UK gas and electricity regulator Ofgem through the Low Carbon Networks Fund and Network Innovation Competition. The project’s overall aim was to monitor and model a typical low voltage network using monitoring in households and substations in order to simulate future realistic demand scenarios. Bracknell, a moderate sized town west of London was chosen as it hosts many large companies and the local network, with its urban and rural parts, is representative of much of Britain’s electricity network.

Fig. 1.8
figure 8

Histogram, logarithmic y scale, and box-plot of half hourly measurements in TVV data

Full size image

This data set contains profiles for 226 householdsFootnote 3 on half-hourly resolution between 20th March 2014 and 22nd September 2015. The measurements for these households are timestamped and as was done for the Irish smart meter data, information of the day of the week, half hour of the week was deduced. We have also added a period of the week which marks each half hour in a week and ranges from 1, corresponding to 00:15 on Monday, to 336, corresponding to 23:45 on Sunday. We have also subset the data to include only full weeks. Thus, in this section, the analysis is presented for observations taken between 24th March 2014 and 20th September 2015, spanning 546 days which is 78 weeks of data.

We again start by considering the histogram and box plot of all measurements (Fig. 1.8). The largest value in this data set is 7.623 kWh, which is much smaller than our last case study, whereas the 75th percentile is 0.275 kWh. Though the magnitudes of these values are not the same, the general shape of the histogram here is similar to that of the Irish smart meter data; they are both left skewed and large values are relatively few.

The box plot presented in Fig. 1.9 shows the consumption for each household.

Next, we consider the general patterns and trends in the load. We do this by considering the average consumption. Let us start with the top left image of Fig.  1.10. Firstly, it shows that measurements were taken 15 min after and before the hour. The mean profile also appears to be less smooth as expected, than in the case of the Irish smart meter data, as they are less households. Still some fundamental and qualitative similarities persist; on average, electricity demand is low at night. This increases sharply after around 6 am and reaches its peak around 7.45 am. This surge in demand stabilises until a small peak during typical lunch time hours. Again, the evening peak is still the period of highest demand; the peak reaches higher than 0.3 kWh and is sustained for roughly 3 h. Note that if a household has electric vehicle, this will change the demand profile. However, as we discussed before, the presence of electric vehicles will change not just the timing of this high demand but also magnitude and duration.

Fig. 1.9
figure 9

Box-plot of electricity load of each household in the TVV data

Full size image
Fig. 1.10
figure 10

Average demand profiles in (kWh) for various time horizons in the TVV data

Full size image

Of course, these values depend on the time of year and the day of the week as shown in the top right and bottom left plots of Fig. 1.10. The seasonal and annual cycle for daily average demand is obvious from the top right plot. Recall that day 1 corresponds to the 20th of March 2014. Although it would be valuable to have an even longer time series, there are some periods for which two consecutive seasons of data are present. This in general helps in forecasting, because it enables modelling seasonal and annual cycles.

The weekly cycle shown in the bottom left plot of Fig. 1.10 is again in line with what we saw with the Irish smart meter data. On average, the weekends have high electricity consumption with lowest average demands being recorded between Wednesday and Friday. It may be possible that Mondays are relatively high because this plot does not differentiate between Mondays which are weekdays and Mondays which are bank holidays. We will consider this shortly.

Finally, the bottom right plot in Fig. 1.10 reaffirms the seasonal cycle; winter months on average have higher electricity demand than do summer months. This is due to increased lighting, but it is also possible that there are at least some houses in the sample that heat their homes using electricity. Note while this seasonality may be important to model when forecasting aggregated load, it may be less important for forecasting individual load (see e.g. Haben et al. [17], Singh et al. [13], where gas heating is more prominent, like in most parts of the UK (see Department for Business, Energy & Industrial Strategy, UK [18]).

While a day may be classified into the day of the week, we may also classify them by whether it is a holiday or working day and whether it succeeds a holiday or working day. Thus, we now consider how the top left plot of Fig. 1.10 changes depending on such a classification.

Fig. 1.11
figure 11

Average demand profiles in (kWh) for each day classification by time of day for TVV

Full size image

The days and times were classified into 4 categories: working day followed by working day (ww), working day followed by a holiday (hw), holiday followed by a working day (wh) and holiday followed by a holiday (hh). All Sundays were classified as “hh” but weekdays can be classified as “hh” for example for Christmas or other bank holidays. Tuesdays to Fridays are mostly qualified as “ww” except when they occur immediate after Easter weekend, Christmas, boxing day, or new year’s day, in which case they were classified as “hw”. As expected, Saturdays are mostly qualified as “wh” or as “hh” when they succeeded Fridays which were national holidays. The load profiles separated by these day classifications are shown in Fig. 1.11.

Again, we see qualitatively similar behaviour as the Irish smart meter data; the breakfast peaks occur earlier on working days and at similar times regardless of whether the previous day was a holiday or not. As was the case for the Irish smart meter data, the evening peaks are not distinguishably different between working days and holidays; the main difference is for day time consumption. In general, bank holidays and Sundays have the highest usage, Saturdays and other ordinary non-working days use slightly less but still significantly more than working days. The day time usage on working days is the lowest.

1.3 Outline and Objectives

As was mentioned before, the work that is presented in this book is the first part of a project, which aims to incorporate analyses of extremes into forecasting algorithms to improve the accuracy of forecasts for low-voltage networks, that is substations, feeders and households. Thus, it is an amalgamation of two research areas, which till now have remained relatively separate, in order to inform and affect decision making within the energy industry.

Thus far, we have considered only generally the value of the current line of inference to the utility industry. In what proceeds, we aim to give a thorough review of the literature and provide more specific reasons for why each method is used and discuss its shortcomings. In Chap. 2, we will explore in depth the literature of short term load forecasts (STLF). Within it, we will consider some industry standards, introduce some recent forecasting algorithms, and discuss forecast validation and uncertainty. After that, we will deviate for two chapters into the theory of extremes (Chap. 3) and the statistics of extremes (Chap. 4), both of which form the cornerstones of the work presented in the case studies in Chaps. 4 and 5. Presented forecasting and extremes techniques are illustrated in case studies. Benchmarks for end-point estimators of electric profiles and forecasting algorithms are established, some modifications offered and crucially analyses of extremes is provided, which in return feeds into forecasts and their validation.