Introduction and background

There is substantial activity surrounding the implementation of the Smart Grid in California and across the nation. Some of the main promises of the Smart Grid include increased system efficiencies, improved reliability, and a better informed, more involved, and happier customer base. On the ground, however, utilities are still struggling to communicate the value of the Smart Grid to their customers. In some places, there have been high profile cost overruns and outright customer revolts with respect to the communicating interval meters. Our goal in this study was to investigate the value of Smart Grid-related information and controls, both to utilities in the form of energy and peak load savings, and to customers in the form of program satisfaction and electricity bill savings.

The first phase of this research effort took place in 2008, when the Sacramento Municipal Utility District (SMUD) piloted an energy-efficiency and demand-response program for small commercial customers with peak demands less than 20 kW. The Small Business Summer Solutions Study, as it was known to participants, offered customers two mutually exclusive program options: (1) a time-of-use rate with critical peak price events, or (2) monthly payment for remote thermostat control by SMUD. Participating small businesses received programmable thermostats that allowed them to program daily schedules, and also to automate response to the 12 event signals broadcasted by SMUD that summer. Results showed that, compared to the previous summer, customers reduced overall energy use by 20 %, shifted 14 % of their load out of the event periods, and saved 25 % on their electricity bills. About 80 % of participants said that the program met or surpassed their expectations, and three quarters said they would probably or definitely participate again without the $120 participation incentive. These results provided evidence that energy efficiency programs, dynamic rates, and load control programs could be used concurrently and effectively in the small business sector, and also provided further evidence of the effectiveness of communicating thermostats for near-immediate air-conditioning demand response (Herter et al. 2009).

Like the Small Business Summer Solutions Study, the 2011 Residential Summer Solutions Study incorporated smart thermostats, a dynamic rate, and a load control option. In addition to these, the residential study also incorporated three different real-time information treatments—Baseline information (no real-time data), real-time Home information, and real-time Appliance information—with the main goal of examining the effects of these information treatments on overall energy use, daily peak loads, and event loads. At the same time, the study was designed to compare the viability of two demand response program types: dynamic peak pricing with customer-controlled thermostat automation versus a load control incentive program with utility-controlled thermostat automation. The pilot program was offered on a voluntary basis, meaning that results are representative of a voluntary program offered to similar customers. The findings of this study are expected to inform SMUD’s future information, control technology, rate, and program offerings for the residential sector (Herter Energy Research Solutions 2012).

Study area

This study takes place in the two cities of Sacramento and Folsom, California. Both cities are located in the SMUD service territory, which is located in California’s central valley, covering the state capital of Sacramento and surrounding suburban areas. Sacramento weather is characterized by mild, rainy winters and hot, dry summers. On average, the maximum daily temperature exceeds 90 °F (32 °C) on 74 days annually, and exceeds 100 °F (38 °C) on 15 days annually.

SMUD is currently the sixth largest community-owned electric utility in the nation, spanning 900 square miles and serving over half a million residential customers. In 2009, SMUD received a U.S. Department of Energy Smart Grid Investment Grant of $127.5M to implement an Advanced Metering Infrastructure (AMI) for all of its customers, and to subsequently implement large-scale demand-response and dynamic-pricing programs. SMUD began installation of their new AMI system at the end of 2009 and was 99 % complete by the spring of 2012.

Experimental design

The Residential Summer Solutions study involved random assignment of three experimental information treatments combined with two self-selected program options. The information treatment groups consisted of a Baseline group that received no real-time information, a Home group that received real-time electricity use and cost data at the whole-house level, and an Appliance group that received real-time electricity use and cost data for the home, air-conditioning (AC) and two additional appliances. In the participation agreement, customers were asked to choose from two programmatic options that determined their incentive for responding to the 12 planned demand response events: the Summer Solutions rate—an experimental time–of–use rate with critical peak events (a TOU-CPP rate)—and the automatic temperature control (ATC) option—an incentive-based AC load control program. These two options created four voluntary program combinations: Summer Solutions rate only, ATC option only, both options, and neither option. Table 1 provides an overview of the Summer Solutions Study design.

Table 1 Summer Solutions Study design summary

Equipment and information treatments

The Summer Solutions equipment was custom designed for this study. The base unit is an Internet-connected gateway with two-way wireless communication via ZWave to local system components, including smart thermostats and electrical submeters. The gateway was designed to retrieve OpenADR event information through the Internet connection and pass it to the thermostats to initiate AC automation during events. The gateways also served to aggregate energy use data from the submeters for presentation on the Energy Display (Fig. 1), accessed through an Internet browser on any device connected to the participant’s local area network, most commonly the home computer. Electricity use and cost information was updated on the Energy Display every 15 s. Summer Solutions thermostats also collected data directly from submeters and displayed instantaneous electricity use for up to four submeters at 15-s intervals.

Fig. 1
figure 1

Energy display interface (© Candi Controls)

The three randomly assigned information treatments necessitated three separate equipment configurations. Participants in the Baseline information group were given a gateway and a smart thermostat. The Home information group received the same smart thermostat plus a single submeter that allowed participants to view the real-time energy data for their home on their thermostats and computers. The Appliance group was given the same thermostat and whole-house data, but was also provided with three additional submeters: one for their AC, one for a 220-volt appliance of their choice, and one for a 110-volt appliance of their choice.

Self-selected rate and thermostat control options

Participants could pick one, both, or neither of the two Summer Solutions program options: the Summer Solutions rate and the Automatic Temperature Control option. Customers not wanting either option could still participate by having the Summer Solutions equipment installed at their homes; these participants received notification of system events, but had no incentive or obligation to respond.

The Summer Solutions rate is a tiered time-of-use rate with 75-cent critical peak price events, commonly referred to as a TOU-CPP rate, chosen in part for its success in dozens of previous electricity pricing studies (see, for example, Faruqui and Palmer 2012). Assuming no price response, the rate is revenue neutral for SMUD’s summer months, from June to September. In the winter months, the standard residential rate applies. On summer weekends, holidays, and outside the 4–7 pm weekday peak period, the Summer Solutions rate is about 30 % lower than the standard rate, providing substantial opportunity for those on the rate to save money by shifting energy use off peak. On normal summer weekdays, the rate increases to 27 cents per kWh during the 4–7 pm peak. On the 12 event weekdays, the 4–7 pm rate jumps to 75 cents per kWh (Fig. 2).

Fig. 2
figure 2

Summer Solutions rate magnet

For comparison, SMUD’s standard two-tier residential rate was 10.45 cents per kWh for the first 700 kWh and 18.59 cents per kWh for any use above 700 kWh in a billing period. Note that the inclining block or tiered rate structure existing in SMUD’s current residential rate was carried over into the experimental rate to avoid penalizing participants that ordinarily use less than 700 kWh per billing cycle. The tier adjustment for the Summer Solutions rate takes place only in the off-peak period, creating a total of four possible prices: Event, On-peak, Off-peak under 700 kWh, and Off-peak over 700 kWh.

The ATC option was offered to all participants, regardless of their rate choice. Participants choosing the ATC option received a $4 credit on their bill for each peak event, during which a 4 °F (2.2 °C) offset was initiated and could be overridden only once per summer. Customers who signed up for both the Summer Solutions rate and the ATC option were thus paid twice for their demand response—once by the rate savings and once by the ATC event incentive.

Field study activities

In the spring of 2011, SMUD invited 7,000 eligible customers to participate in the Summer Solutions study. In addition to the state-of-the-art real-time energy information system and thermostats, customers were offered:

  • Lower energy bills through efficiency improvements, conservation, or by taking advantage of the Summer Solutions rate’s 30 % off-peak discount;

  • $4 bill credit for each of the 12 summer events, for those choosing the ATC option;

  • Free Home Energy Assessments, along with on-site, personalized advice and a list of inexpensive efficiency improvements, specific to the home, that the participant could complete themselvesFootnote 1;

  • Goodwill—for many customers, participation in a program like this is about feeling that they are doing the right thing: benefiting the system, the environment, and the community.

Of the 7,000 invited customers, 446 submitted signed participation agreements. Customers were accepted in sequential order. Of the first 309 customers contacted for installation, 16 changed their mind during the installation process, 26 homes were not compatible with the technology, 2 dropped out after installation, and the remaining 265 were participants throughout the summer. The 137 customers who signed up but were not contacted for installation were assigned to a wait list for participation in 2012. The final sample sizes for the Summer Solutions study are provided in Table 2, indicating the three information treatment groups down the left side, and four program option groups across the top.

Table 2 Experimental sample

The Summer Solutions equipment installations began as soon as participation agreements were received and were completed by June 2011. These were not simple installations. The required skill sets included those of a licensed electrician, to install the submeter current transformers in the service panel, those of an air-conditioning technician, to install and test the thermostat, and those of a network technician to connect the gateway to the home network. Furthermore, installation was complicated by three different equipment configurations and four different program options. To facilitate timely and accurate installation, installers were provided with a detailed checklist and information specific to each participant.

Educational and customer support

During the installation appointment, program materials were physically handed to participants. All participants were provided with a Quick Start Guide, tailored for their particular equipment and program. Those on the Summer Solutions rate also received a refrigerator magnet illustrating the basics of the new pricing (Fig. 2). At the end of the installation appointment, the installer worked with participants to schedule the thermostat as desired, and provided a brief tour of the thermostat programming options.

Customer support was provided by the “Summer Solutions Team” through a dedicated phone line and email address. The Summer Solutions Team handled all lower-level billing, programmatic, and technical questions, and delegated higher-level issues to SMUD or equipment technicians as needed. By the end of the summer, 138 service calls had been made, or roughly one site visit for every two participants.

In addition to personal attention provided by the installers and the customer service team, participants were provided several opportunities to review information about the pilot and actions they could take as participants. A website was created to provide basic information about the pilot, including links to participant materials and surveys, frequently asked questions, a discussion board, equipment information, rates information, links to rebates and energy saving tips, and customer service contact information. Email notification 1 day prior to each of the 12 events throughout the summer reminded participants of specific strategies that could be taken to reduce energy consumption during the event on the following day.

Throughout the summer, participants were encouraged by email and through the participant website to receive a free Home Energy Assessment. The assessment consisted of a 30–60-min visit from a trained home energy survey provider, who then provided customers with pictures of problem areas and a short list of simple, low cost actions they could take to improve the efficiency of their home. By the end of the summer, one third of participants had taken advantage of this offering. Although these assessments were done at low cost and did not provide services or measures beyond advice, those receiving assessments rated the service highly, and had significantly greater peak energy savings than those who did not receive the service. Overall energy savings were also greater, but the impact was not statistically significant.

Events

The summer of 2011 saw only 3 days with temperatures above 100 °F—far short of the typical 15 that the Sacramento area typically gets. Despite the cool summer, 12 events were called on days with maximum temperatures forecast to be greater than 95 °F, trying to keep a good distribution of event days and non-event days across the available temperature spectrum and days of the week.

Participants were given 24-h advance notification of impending events, via email, Energy Display, thermostat display, and, if requested, by phone call or text message. For each event, the notification email provided new tips for responding to the event on the following day. At 4:00 pm on the day of the event, thermostats were designed to respond to the OpenADR notification by initiating the event setting. The default event setting for all Summer Solutions thermostats is 4 °F (2.2 °F) higher than the setpoint at the time of event initiation. Participants that chose the ATC option could not change this default value, and could override it just once per year. Non-ATC participants could change the default setting at installation or any time afterwards, and could override as many events as desired.

During the summer 2011 test period, the OpenADR communications platform successfully initiated all 12 events. The Summer Solutions equipment effectively initiated event response for most participants, but at any given time about 40 gateways (15 %) were not connected to the Internet and so were incapable of receiving the event signals.

Billing

All participants continued to receive their standard SMUD bill. Bills for participants on the ATC option included a line item indicating the Summer Solutions ATC payment of $4 per event for that billing period. Bills for participants on the Summer Solutions rate included a separate Bill Comparison sheet that outlined how the Summer Solutions rate reduced or increased the amount they owed relative to the standard rate. In the post-summer survey, this rate comparison sheet turned out to be one of the most popular features of the Summer Solutions Study.

Data collection

Multiple types of information were collected from study participants at several points in the project. Basic customer information was pulled from SMUD’s customer database to conduct recruitment efforts. More detailed customer and building information was collected from participants at the beginning of the summer through the Participant Survey. Throughout the study, SMUD collected hourly electricity use data using their new AMI system. At the end of the study, participant perceptions of the program were documented in their End-of-Summer Survey answers. Of the 265 participants who completed the study, 245 (92.5 %) completed the Participant Survey and 251 (95 %) completed the End-of-Summer Survey.

Results

Hourly kilowatt data were analysed at the individual customer level using mixed-effects models, also referred to as hierarchical or multilevel models. These models provide a major statistical framework for many kinds of statistical problems involving nested data and are now routinely applied in a great variety of disciplines. Thus, there is an extensive statistical literature on this particular methodology. (See for example Raudenbush and Bryk 2002.) For the current study, a sample of customers was obtained, and from those individuals, hourly data were obtained. These data have a hierarchical structure: hourly kilowatt data are nested within days and days are nested within customers.

For each analysis described below, three types of impacts were calculated: Overall Energy impacts, Non-event Peak impacts, and Event Peak impacts. These three values represent different time periods, as follows.

  • Overall Energy impacts represent the average change in hourly energy use across all 24 h of the day and all 7 days of the week, including weekends and holidays;

  • Non-event Peak impacts represent the average hourly change across the three peak hours, from 4 to 7 pm on non-event weekdays (weekends and holidays are excluded);

  • Event Peak impacts represent the average hourly change across the three event hours, from 4 to 7 pm on event weekdays.

The analysis measures all impacts relative to baselines modeled using summer 2010 participant loads corrected to reflect 2011 temperatures. The same analysis applied to the wait list, used as a control group, showed no statistically significant difference between 2010 and 2011 energy use; thus, a correction for exogenous effects was not required. For consistency and ease of comparison, all impacts are presented in units of average kilowatt-hours per hour, abbreviated in most cases to kilowatts. Positive impact values indicate an increase in energy use relative to the baseline, whereas negative values indicate energy savings.

Load impacts: effect of real-time information

Although this study touches on many aspects of new residential programs and technologies, the main objective of this study was to investigate the usefulness of real-time energy data at the home and appliance levels in reducing energy use and peak loads. This objective is met through a three-way comparison between the Baseline information group, the Home information group, and the Appliance information group.

Table 3 provides the load impacts for each of the three information groups. In each case, the negative kilowatt values indicate the average hourly savings for the summer of 2011, where each group is compared to its own 2010 baseline corrected for weather effects.

Table 3 Average hourly load impacts, by information group

Keep in mind that the Baseline group received the same non-treatment interventions as the Home and Appliance groups: invitation to participate, website, customer support, home energy assessments, and the like. Thus, a direct comparison of the information treatment groups to the Baseline group negates the effects of these non-treatment interventions, revealing the effect of the information treatment alone.

Compared to the Baseline group, real-time Home information lowered overall energy use by about 4 %. This value translates to 50 W savings every hour of the summer, or roughly 150 kWh savings over the course of a 4-month summer. This energy savings effect was not seen for the Appliance information group.

Real-time information at both the Home and Appliance levels had a significant effect on non-event peak loads relative to the Baseline group, with real-time Home information increasing savings by 4 %, and real-time Appliance information increasing savings by 7 %.

In contrast, real-time information did not have a significant effect on event peak impacts—i.e. all three information treatments dropped roughly 1 kW during events. Figure 3 illustrates the variation in impacts by treatment group.

Fig. 3
figure 3

Load impacts by information treatment (ΔkW). Values in bold are statistically different from the values for the “Baseline information” group

Load impacts: effect of dynamic rate and ATC

Unlike the information treatments, which were randomly assigned to customers prior to recruitment, the Summer Solutions rate and ATC option were offered in the Participation Agreement, where participants could freely choose to sign up for the Summer Solutions rate only (SS Rate), the ATC option only, or both, or neither. As a result, the reader is cautioned against extrapolation of any individual program option result to the general population; however, results are representative of a voluntary program in which the same program options are offered. Load impacts by program option are shown in Table 4.

Table 4 Average hourly load impacts, by information group

In every case, the “Neither option” group—participants who chose to have the equipment installed, but chose to not sign up for either the Summer Solutions Rate or the ATC option—was the least responsive of the four groups. Overall energy savings were similar for the three groups that chose at least one program option; however, peak impacts varied significantly. On both event and non-event weekdays, those on the Summer Solutions rate saved significantly more than did those on the ATC program only. Figure 4 illustrates the variation in impacts by treatment group.

Fig. 4
figure 4

Load impacts by Summer Solutions rate and ATC options (ΔkW). Values in bold are statistically different from the values for the “Neither option” group

Bill impacts

On average, Summer Solutions participants saved about $10 (7.6 %) per month on their electricity bills due to energy savings alone. For those on the standard rate, bill savings ended there. Those on the Summer Solutions rate saved an additional $9.45 per month on average, for a total savings of roughly 15 % compared to what they would have paid had they not signed up for the study.

Figure 5 shows a scatterplot of rate-dependent bill impacts for those on the Summer Solutions rate. Nearly three quarters (72 %) of the participants who signed up for the Summer Solutions rate saved money on their 2011 summer bills relative to the standard rate. At the extremes, about 4 % of participants saw their summer bills increase by more than 10 % as a result of the rate, while one third (33 %) saw their summer bills decrease by more than 10 % as a result of the rate.

Fig. 5
figure 5

Bill impacts participants on the Summer Solutions rate (US dollars vs. percentage)

Participation rates

One of the secondary objectives of this study was to estimate participation rates for the Summer Solutions rate and ATC programs. Of the 237 customers that responded to the first 4,000 invitations, 117 (49 %) chose both the Summer Solutions rate and the ATC option, 60 (25 %) chose the Summer Solutions rate only, 31 (13 %) wanted only the ATC option, and 30 (13 %) wanted neither option. The overall 75 % signup rate for the residential Summer Solutions rate is comparable to the 65 % signup rate found for the TOU-CPP rate offered in the Small Business Summer Solutions Study (Herter et al. 2009).

Correlating behaviour with load impacts

This section reports the Pearson product–moment correlation coefficients (r) for correlations between customer-specific energy use, impact values, and survey answers collected after the summer 2011 test period. Customer-specific impact values for these correlations were calculated using a difference-in-differences between treatment and control group 2010 and 2011 values.

Table 5 provides correlation coefficients for customer-specific impact values and summer 2011 behaviours collected via the post-summer survey. Values are statistically significant at the α = 0.05 level where |r| ≥ 0.14, presented in bold font for ease of review. Notable results include significant savings across the board for participants who set their thermostats at 78 °F (25.6 °C) or higher in summer, or increased their thermostat setpoint during the peak period. Participants who precooled before the peak period saved on both non-event and event peak periods. Overall energy savings were improved for participants who replaced older AC units. Non-event peak savings were significant for those who removed a refrigerator from the garage or had a Home Energy Assessment completed. Load sheds during events were also improved for those who spent event periods outside the home and for those who avoided showers because they have electric water heaters.

Table 5 Correlation between impacts and load reduction behaviours (R)

Thermostat offsets and overrides during events

At the end of the summer, the Summer Solutions thermostats were polled for user programmed offsets and event overrides. Recall that customers in the ATC group had an obligatory 4 °F (2.2 °C) event offset, while all non-ATC participants had the option to change this offset to any value 0–10 °F (0–5.6 °C). Among those on the Summer Solutions Rate, very few changed the default setting, resulting in an average programmed offset of 3.9 °F (2.2 °C). The non-ATC participants on the standard rate were most likely to downgrade their event offset, but even this group maintained an average default event offset of 3.3 °F (1.8 °C; Fig. 6).

Fig. 6
figure 6

Average event offset, by rate and ATC participation

An event override is an intervention by an occupant to change the preset automated response during an event period. Participants on the ATC program were allowed to override 8.3 % of summer events (1 out of 12), while non-ATC customers on the Summer Solutions or Standard rates were allowed to override 100 % of events. Figure 7 shows that overrides were most prevalent for those who had no monetary or obligatory incentive to respond to events, i.e. the Non-ATC participants on the Standard rate. On average, those on the Summer Solutions rate overrode less than one of the 12 events, while those on the ATC program were least likely to override.

Fig. 7
figure 7

Event overrides, by rate and program

Participant satisfaction and comments

Of the 236 participants who rated the Summer Solutions Study in the End-of-Summer Survey, 83 % said their satisfaction with the program was Excellent or Good, and more than 90 % of participants chose to remain in the Summer Solutions Study for 2012. Of those who dropped, 5 % explicitly chose not to participate beyond 2011, and 5 % were unreachable.

Conclusions and discussion

SMUD’s 2011 Residential Summer Solutions study tested the effects of real-time electricity data on customer behaviour and hourly electric loads in 265 Sacramento-area homes. As part of the study, a voluntary dynamic rate and payment for event-based air conditioning control were offered, and a second analysis considered the differential effects of these program options on the loads and behaviour of the self-selected groups. The study touched on a broad spectrum of utility issues including marketing, education, rate design, meter technology, information displays, home networking, customer service, consumer behaviour, conservation, efficiency, load shifting and demand response to name a few. Table 6 summarizes the key findings of the study. The following sections provide further insights based on the totality of this effort.

Table 6 Summer Solutions Study results summary

Real-time energy information

The results of this study do not support the provision or rebate of real-time energy monitors for individual appliances, due to the high cost, limited energy savings, and lower customer ratings for this feature. These results may change as home information and automation systems improve and a younger group of customers become homeowners. The results presented here show that real-time Home data enhanced overall energy savings, while real-time Appliance data did not. One possible explanation for this pattern is that those with Home information employed a more general strategy for keeping all energy use low all the time, while those with Appliance energy data were more strategic in their load management efforts, e.g. those with appliance-level cost information might make rational economic choices to continue using the service when they see the relatively low cost of doing so. Given the uncertainties here, this topic may warrant further research.

Neither type of real-time data affected response to events. Here, thermostatic automation may be playing a role by removing the need for customers to pay attention and respond to information during events. Following this line of reasoning, future research might address the following questions:

  • If thermostatic automation for non-event weekday peaks were made easier, would real-time information effects for the weekday peak period be diminished or disappear?

  • If the thermostats provided energy efficiency optimization, would real-time information effects on energy savings be diminished or disappear?

  • If the answers to both of the above questions are affirmative, might it be possible to achieve the majority of benefits attributed to real-time energy information (behavioural response) through better thermostat design (automated response)?

Dynamic rates vs. load control

Grossly speaking, the demand response field tends to be comprised of two opposing groups: those who believe that customers will find dynamic rates unpalatable because they are confusing and punitive, and those who believe that customers will find direct load control unpalatable because it usurps customers’ control over their own appliances. This study found neither claim to be true. Given the choice between a TOU-CPP rate, a direct load control program, both, or neither, half of participants chose the combined rate and load control, one quarter chose the rate only, and the remaining quarter was split evenly between load control only and neither option. Also of interest was the fact that only one of the 265 participants indicated that no one would be home between 4 and 7 pm on weekdays, despite the obvious incentive for such customers to sign up for a free benefit.

The results of this study do not support the use of direct load control where dynamic rates are an option. On average, customers on the ATC program received a monthly benefit of $12 for a 0.87-kW (36 %) event response and 1.2 % override rate. In contrast, customers on the Summer Solutions rate benefited by $9.45 per month for 1.18-kW (53 %) event response with an override rate of 6 %. Despite the lower override rate of the ATC group, their event savings lagged that of the Summer Solutions rate group, implying that those on the Summer Solutions rate either increased the event offset above the default or contributed to response with non-AC loads. In addition, these results imply that dynamic pricing has the potential to provide utilities with more demand response at lower cost than direct load control programs.

Technology

While the Summer Solutions equipment worked well enough in aggregate, as evidenced by the significant energy and demand savings, the system did not work as intended in all homes all of the time. About 7 % of meter panels were unable to accept the current transformers needed to provide occupants with real-time data for their home. One would expect resolution of this issue once meters transmit the data directly inside without the help of sub-meters. Throughout the summer, almost all of the service calls (93 %) and 138 technical support site visits were directly related to equipment problems, with networking problems being the main culprit. These numbers call into question whether these findings are evidence of a promising industry’s early growing pains, evidence that such complex systems are likely to be impractical and cost-ineffective for a large number of homes indefinitely, or simply one unreliable implementation of an otherwise effective technology. Over the next decade, as standards converge, technologies improve, and “real-time energy information” becomes a household term, we expect that the answer to this question will become better understood.

The OpenADR communications technology successfully notified the Summer Solutions thermostats within seconds of sending the OpenADR signal—as long as (1) the gateways were connected to the Internet and (2) the ZWave connection between the gateway and thermostat was robust. In other words, where the event notification did not reach its intended target, OpenADR was not at fault, but rather, the equipment was.