Blog Posts

Hourly Aggregation Approaches and Uncertainties

6/5/2018

Week Seventeen CalTRACK Update

Establishing guidelines for aggregating building-level hourly energy savings into portfolio loadshapes requires careful consideration of information and uncertainty-level preferences for various use cases. The issues with aggregating energy savings for hourly methods and potential solutions are outlined below:

Aggregation Method for Billing Period and Daily Methods:
In daily and billing period methods, building-level savings are generated by summing the building’s estimated energy savings for each day or billing period of the reporting period. The portfolio savings are then calculated by aggregating total savings for each building in the portfolio.

Why are daily and billing period aggregation methods problematic with hourly models?
For hourly models, portfolio uncertainty is difficult to calculate when savings are aggregated for each hour due to correlation in the error term.

Suggested Hourly Aggregation Methods:
To begin our discussion of hourly aggregation methods, consider two types of roll-ups:

Vertical Roll-Ups:
In a vertical roll-up, hours within a day are grouped together for each building before aggregation. For example, one may choose to aggregate hourly energy savings in three-hour intervals throughout the day instead of each hour individually.
Although vertical roll-ups can reduce portfolio uncertainty, larger time intervals provide less information in portfolio loadshapes. Hourly methods are created to provide granular information about energy load impacts during each time-of-day. Less information is available if hours are “rolled-up” into larger time intervals.

Horizontal Roll-Ups:
Horizontal roll-ups aggregate each hourly estimate with estimates of the same hour across weeks. A horizontal roll-up can aggregate individual hours or time intervals, such as the three-hour interval discussed above.

Other Considerations:
There were a few additional suggestions from the working meeting (5/24) that could help create guidelines for aggregating portfolio loadshapes:

Take 8760 (full year hourly load) building data and establishing criteria for “slicing and dicing” the data to identify patterns that can inform hourly aggregation guidelines.
Leveraging measure specific load shapes, from existing technical reference manuals or deemed savings models, was yet another idea that was brought to the table.

In the coming weeks, we welcome proposals and testing criteria to determine appropriate guidelines for utilizing the vertical and horizontal roll-ups to aggregate hourly savings into loadshapes.

Homework:

Provide proposals and testing criteria for aggregating portfolio loadshapes on GitHub (issue 97)
Attend standing meeting on 6/7 at 12:00 (PST)
Contribute to the Sand Box of future issues

0 Comments

Site-Specific Hourly Methods Finalized

5/30/2018

1 Comment

Week Sixteen CalTRACK Update

During the standing meeting on 5/24, the working group finalized hourly methods for calculating hourly energy savings and commenced discussion on aggregating hourly savings into portfolio loadshapes. The finalized hourly methods and an introduction on aggregating portfolio loadshapes are outlined below.

View May 24, 2018 Working Group Meeting

Finalized Hourly Methods:

Data sufficiency for independent variables will be defined by coverage instead of the minimum time period required for daily and billing period methods.
The temperature variable will be defined by fixed temperature bins between 30-90 F instead of the variable degree day balance points used in billing period and daily methods.
In Lawrence Berkeley National Lab’s (LBNL) TOWT model, there is a model adaptation function that gives higher weight to recent data to improve short-term demand forecasts. This model adaptation function will not be used in CalTRACK’s hourly methods.
The occupancy variable will be defined with LBNL’s default occupancy algorithm. This is described in greater detail in Phil Price’s Everything I Know About Building Energy Modelling, But Never Told Anyone Before (18:30-30:00).

Aggregating Hourly Savings into Portfolio Loadshapes
To provide an accurate valuation of energy efficiency as a grid resource, energy savings must be quantified at specified time intervals and geographic locations. To create portfolio loadshapes, building-level savings must be aggregated. The method of aggregation has implications on the portfolio uncertainty and provides different granularity of information for aggregators, utilities, and customers. Different use cases may prefer different aggregation methods based on priorities specific to their use case. To accommodate different use cases, flexible methods for aggregating hourly savings into portfolio loadshapes may be preferred.
As we explore this topic further, some potential use cases to consider are:

Pay-for-Performance Programs
In the PG&E pay-for-performance program, the utility provides incentives for peak savings. This requires estimates of portfolio savings at the hourly level.

Non-Wires-Alternative Procurement
Non-Wireless-Alternative procurements require estimates of portfolio savings for buildings connected to specific grid nodes in order to measure grid impacts and potentially avoid infrastructure investments.

Cap and Trade, Greenhouse Gases, or Carbon Tracking or Trading Initiatives
Initiatives attempting to accurately quantify carbon offsets from energy efficiency investments require savings estimates at specified time and geographic locations because generation portfolios utilize resources with different carbon intensity at different times and locations.
We discussed a few options for aggregation methods, and look forward to input from the working group in the coming week.

Homework:

Review final Methods documentation and provide comments on GitHub (issue 101)

Review proposed Hourly guidelines on GitHub (issue 85)
Provide ideas on aggregation approaches for Portfolio Load shapes on GitHub (issue 97)
Contribute to the Sand Box of future issues.

1 Comment

Time of Week and Temperature Model CalTRACK Application for Site Specific Hourly Savings

5/14/2018

0 Comments

Week Fourteen & Fifteen CalTRACK Update

Review of hourly method proposals continued in week fourteen of CalTRACK 2.0 and will be finalized at the 5/24 working group meeting. Lawrence Berkeley National Lab’s Time-of-Week Temperature (TOWT) model is the specification to be used in CalTRACK 2.0. Mathieu et al. describe the application of TOWT models in Quantifying Changes in Building Electricity Use, with Application to Demand Response.

Overview of TOWT models:
As energy efficiency finds its legs as a grid resource, time dependent savings will be essential to the value proposition. Pay-for-performance programs can leverage this value with accurate building-level energy savings calculations at granular time intervals. TOWT models are one method for calculating energy savings at the hourly level.

Strengths:

Aggregated portfolios for pay-for-performance programs may have high variability in hourly energy consumption due to its nature. This variation is evident in figure 1, which shows load patterns over time for an office building, furniture store, and a bakery. The TOWT model addresses this variability in the following manner:
An “occupancy” proxy is determined using a linear regression model and allows for the hourly data to be segmented based on a building’s occupancy status.
Several “time-of-week” independent variables (one for each hour of the week) are included in the main linear regression model to capture hourly load variation. For example, a restaurant may regularly consume more energy on Friday nights because the restaurant has more customers on Friday nights. This type of variation will be controlled for by the “time-of-week” covariate.
The temperature covariate uses 7 bins of fixed temperature ranges instead of employing a grid search to find the balance points. Due to higher amounts of data in hourly methods, the fixed temperature ranges provide a simpler solution without significant drawbacks.

Weaknesses:

By nature, calculating hourly energy savings requires more granular data. This can make data sufficiency problematic.
Similar to daily methods, energy consumption on weekends or holidays may be different than typical days.
There is autocorrelation in the errors of parameter estimates, which complicates uncertainty calculations.

Homework:

Review draft of billing and daily methods write-up
Review proposals for hourly guidelines on GitHub
The next working group meeting is on Thursday, 5/24

0 Comments

Considering Hourly Methods: Data & Use Cases

5/7/2018

0 Comments

Week Thirteen Update for CalTRACK

During week thirteen, the CalTRACK working group discussed proposals for hourly methods in the standing meeting. The discussions included helpful suggestions of other reference materials as well as variations that may be appropriate for different applications of hourly methods and suggested improvements in CalTRACK 2.0’s documentation. The video from May 3, 2018 is provided at the end of this post.

Hourly Methods
In the development of hourly methods, the goal is to establish guidelines that mirror the methodology in billing period and daily methods. However, hourly methods have unique complexities that require departures from billing period and daily methods. These complexities are identified and discussed below:

Data Management
When compared to hourly data, daily and billing period savings calculations have higher data sufficiency requirements because hourly data contains more information per time period. This characteristic of hourly data supports the two adjustments to hourly data sufficiency requirements listed below:

 1. Usage data sufficiency will be specified in terms of data coverage (or common support) instead of a minimum time period:
Daily and billing period data sufficiency requirements impose a minimum quantity of time observed from a year of data. In hourly methods, usage data sufficiency will be specified in terms of data coverage in the independent variables. In Time of Week and Temperature (TOWT) models, the independent variables are temperature and occupancy. Data sufficiency requirements will be based on LBNL recommendations for data coverage.

2. Missing Data
Temperature has less variation between hours than days or billing periods. Smaller temperature variation between hours increases the likelihood that interpolated temperature values are accurate. For this reason, interpolated temperature values will be allowed in the reporting period for hourly methods. The threshold of allowable interpolated hours will be determined through empirical testing.

Recommendation for Pay-for-Performance Use Case:

In the baseline period, it is recommended to drop hours with missing temperature and usage data.
In the reporting period, a maximum of 6 missing values per day can be interpolated while maintaining minimum data sufficiency requirements.

Time Of Week Temperature (TOWT) Modeling Approach
The TOWT model, originally by Lawrence Berkeley Lab, contains two covariates:
Occupancy:
Occupancy is an indicator variable that takes the value of 1 if the building is occupied in the hour and 0 otherwise. In LBNL’s model, occupancy of a building is defined by:

Using ordinary least squares regression to establish a regression model for a building.
Grouping all observations at each hour. If 65% of the observations for an hour are above the established regression line, that hour is designated as occupied.
If the condition in (2) is not met, then the hour is defined as unoccupied

Temperature:
The TOWT model allows user-defined temperature bins for modeling a building’s weather dependence. We are recommending setting 7 fixed bins with endpoints at 30, 45, 55, 65, 75, 90, in order to cover a wide variety of climate conditions.

Use Case and Uncertainty

Time-aggregated Uncertainty:
In the program evaluation use case, an analyst may be interested in obtaining time-aggregated savings and uncertainty. Due to residual autocorrelation at the hourly level, aggregating hourly uncertainty for larger time intervals creates imprecise standard errors and uncertainty calculations. Instead, we recommend using daily methods with improved ASHRAE or Ordinary Least Squares (OLS) formulations of Fractional Savings Uncertainty (see Koran 2017) for aggregating uncertainty over time periods.

Hour-level Uncertainty Estimates:
For the procurement and pay-for-performance use cases, regression analysis is an effective tool for acquiring point estimates of savings and uncertainty at each hour. If each building is assumed to have independent errors, the uncertainty at each hour for all buildings in the portfolio can be aggregated without an autocorrelation problem.

Methods Documentation
Currently, the documentation for CalTRACK 2.0 is being updated. The first half of CalTRACK 2.0’s documentation will be posted on GitHub to allow the working group to review and comment on the changes in documentation.

Similar to the methods, the development of effective documentation is an iterative process. The documentation for CalTRACK 2.0 will improve by dividing into three distinct documents:

1. Methods
This document outlines the methodology for quantifying billing period, daily, and hourly energy savings while maintaining CalTRACK-compliancy. In CalTRACK 2.0, the Methods will be organized with a numbering system that corresponds to the Methodological Appendix. This will make referencing and accessing the appendix easier.

2. Methodological Appendix
The Methodological Appendix summarizes discussions and empirical testing that justify methodological decisions. The Methods will reference sections in the Methodological Appendix for readers to easily access empirical support for methodological decisions.

3. Field Guide
A document with minimum requirements for an implementation to maintain CalTRACK-compliancy. This is designed to be a practical and accessible checklist for analysts and other implementers of CalTRACK.

Ideas for Future CalTRACK Work
A sandbox has been added to the GitHub site to document proposals for participants to add ideas for future CalTRACK iterations. If you have an idea for CalTRACK 3.0, or beyond that cannot be addressed this year, please add it the the sandbox.

Additional Hourly Methods Resources

LBNL R Code on Time of Week and Temperature (TOWT)
2002 ASHRAE Guideline 14: Section 5 is relevant for data sufficiency requirements.
Uniform Methods Project – US Department of Energy
1. Peak Demand and Time-Differentiated Energy Savings Cross-Cutting Protocol
2. Whole-Building Retrofit with Consumption Data Analysis Evaluation Protocol

Homework

Review draft of billing and daily methods write-up
Review proposals for hourly guidelines on GitHub
The next working meeting is after 3 weeks on 5/23

0 Comments

Hourly Methods Discussions Continue

4/30/2018

0 Comments

Week Twelve CalTRACK Update

During week twelve, we continued our discussion of hourly methods. In the upcoming week, we will analyze test results for hourly methods on GitHub. We will also be talking about how to log ideas for future improvements. The standing meeting to discuss hourly methods will be on Thursday, May 3rd at 12:00 (PST).

Homework:

Use Hourly Method Tools on test data
Report findings on Github
Offer test criteria for hourly models
Attend standing meeting on Thursday, May 3rd at 12:00 (PST)

0 Comments

Hourly Methods Approach & Testing Considerations

4/23/2018

1 Comment

Week Eleven CalTRACK Update

Week eleven was the first week of hourly methods discussion. Developing hourly methods will require discussion and empirical testing of topics unique to hourly methods before we can make final specifications.
Topics that must be addressed include:

Data sufficiency requirements
Modeling approach
Model selection criteria
The effect of aggregating hourly models on portfolio uncertainty

Time Of Week Temperature Models (TOWT):
A proposed model for hourly methods is the TOWT model from Lawrence Berkeley National Labs (LBNL) is shown below.

Notes:

The number of temperature ranges and the balance points will need to be defined. The LBNL model had 5 ranges ( < 55F, 55-65 F, 65-75 F, 75-90 F, 90 F <).
The methods for defining if a building is occupied or not is explained in detail in Phil Price’s Everything I Know About Building Energy Modeling, But Never Told Anyone Before (18:30-30:00).

Homework:

Use Hourly Method Tools on test data
Report findings on Github
Offer test criteria for hourly models

1 Comment

Building Qualifications Results & Recommendations

4/17/2018

1 Comment

Week Ten CalTRACK Update

The CalTRACK working group finalized discussions on Building Qualifications during the first half of Thursday’s (4/12) meeting and dove into the hourly methods during the second half. Bill Koran from SBW consulting provided a helpful overview of hourly models and the ECAM energy data analysis tool. The major findings of this meeting are summarized below:

Video of 4/12/2018 Meeting

Building Qualification Observations and Recommendations:
Main observations that were driving the recommendations:

Stricter building-level thresholds do not necessarily result in more confidence in portfolio savings.
Thresholds would vary by data granularity. Distributions of building-level metrics also vary widely by building type, location and climate zone.
Bias must be minimized for portfolio aggregation to work.

The building qualification recommendations are dependent on use case:

For use cases where confidence in portfolio-level performance is required (e.g. aggregator-driven pay-for-performance, non-wires alternatives (NWA) procurements), we recommend using a permissive building-level CVRMSE threshold (100% is recommended as a default), but requiring that a portfolio-level metric be respected (e.g. portfolio fractional savings uncertainty).

The portfolio-level threshold will be a policy decision and may differ depending on the use case (e.g. NWA procurement may require less than 15% uncertainty, regular pay-for-performance program may require 25% etc.)

For use cases where confidence in individual building results is required (e.g. customer-facing performance based incentives), ASHRAE Guideline 14 thresholds may be used.

Notes:

The FSU thresholds will be context-specific because the value of certainty is not uniform across projects and procurers. For example, a utility may be willing-to-accept a portfolio with 25% uncertainty in the context of a pay-for-performance program. The same utility may require 15% uncertainty for a non-wires alternative project.

CVRMSE and FSU are different metrics andare described in this paper: (A Comparison of Approaches to Estimating the Time-Aggregated Uncertainty of Savings Estimated from Meter Data)

Goals for Hourly Methods:
Hourly models are necessary for estimating the load impact of energy efficiency. This makes hourly energy savings important information for aggregators and utilities in an energy efficiency marketplace.
Our goal is to establish suitable methods for calculating whole building hourly energy savings for residential and commercial buildings. Additionally, the methods will include guidelines for aggregating site-level savings.

Discussion Topics:
We have allotted three weeks to test and discuss hourly methods. Below are some important topics that will need to be addressed:

Data Requirements and Sufficiency:
- Hourly weather data can be unreliable. We will need to establish guidelines for hourly weather data sufficiency and methods for accommodating missing weather values.
- Data sufficiency requirements for hourly meter data are needed.
- Hourly data cleaning methods must be defined.
Methods:
- It may be beneficial to adjust our model selection criteria for hourly methods.
- Aggregating many hourly models will result in high portfolio uncertainty. As a result, it may be necessary to reassess the calculation of portfolio uncertainty in hourly savings.
Models:
- We will need to determine the most suitable hourly model for CalTRACK. Two open source models to begin the discussion are:
  1. The TOWT model (time-of-week and temperature) from Lawrence Berkeley National Lab
  2. ECAM model from SBW consulting

We look forward to future discussions on GitHub regarding these topics.

Homework:

Use Hourly Method Tools on test data
Report findings on Github
Offer test criteria for hourly models
Watch Phil Price Video about energy modeling and hourly methods: Everything I Know About Building Energy Modeling, But Never Told Anyone Before

1 Comment

Building Qualification Criteria Discussions

4/9/2018

1 Comment

Week Nine CalTRACK Update

During week nine of CalTRACK, there were continued discussions on building qualification criteria and some introductory comments on hourly methods. The working group meeting will be held on Thursday, April 12th at 12:00 (PST). We will discuss:

Final comments on building qualification
Proposals for hourly methods testing

Hourly Methods Overview:
Hourly methods are a new addition in CalTRACK 2.0 and were not in the first version of CalTRACK. This task involves testing various hourly modeling methods and recommending a standardized approach to hourly modeling, which can reveal the time value of energy efficiency.
Importance of Hourly Methods:

Hourly methods are a more granular time interval than daily or billing period methods. These granular time intervals are valuable for determining the temporal value of energy savings, by associating savings with grid demand and energy price.

Time Of Week and Temperature Models:
To start the discussion of hourly methods, it is helpful to consider to existing open source tools. Lawrence Berkeley National Lab has developed an open source time of the week and temperature (TOWT) model to calculate hourly energy savings and is part of the RMV2.0 - LBNL M&V2.0 Tool. A TOWT model predicts hourly energy savings by utilizing hourly temperature data instead of daily or billing period HDD and CDD. Another is ECAM (ENERGY CHARTING & METRICS) developed by Bill Koran at SBW consulting.

We look forward to discussion on the working group call about these approaches, and more to frame the approach to empirical tests for the CalTRACK 2.0 hourly methods.

Final Note:
It is important to remember that CalTRACK methods development is an iterative process. The finalized methods for CalTRACK 2.0, just like v. 1.0, will benefit from field deployment and may need to be revised in future years. We expect this process to result in improved and refined methods with successive iterations.

Homework:

Review final results for building qualifications and provide final edits
Review existing tools, practice and concepts for hourly methods
Discuss and provide suggestions for hourly methods on GitHub
Attend the bi-weekly meeting on Thursday April 12th at 12:00 (PST)

1 Comment

Building Qualifications Test Reveals Wide Applicability of CalTRACK Method for Portfolio Analysis

3/29/2018

2 Comments

Week Eight CalTRACK Update

Today we had an exciting working group meeting focused on Building Qualifications with test results and recommendations. We will be stepping into hourly methods in the upcoming week.

3/29 CalTRACK Working Group Meeting Recording

Daily and Billing Period Methods Specifications:
The final comments on daily and billing period methods have been received. The new, finalized specifications are currently being updated you can track the final on GitHub Issue #82. Participants can review the final specifications and track any issues that fed into the final specification prior to their publication at this site.

Building Qualifications:
The CalTRACK methods are equipped to normalize for weather’s effect on a building’s energy consumption. The methods become unreliable when a building’s energy consumption patterns are instead correlated with non-weather variables. For example, irrigation pumps are used on an agricultural cycle rather than in response to temperature changes. It is reasonable to expect higher energy consumption during the growing season and an analyst may suggest re-specifying the model to control for these seasonal variations. Without modification, existing billing and daily CalTRACK models cannot accommodate these nuanced cases. For this reason, buildings that are not well-specified by the CalTRACK model should be identified and removed from portfolios because they have significant effects on portfolio uncertainty. This issue can be tracked on GitHub Issue #71.

Building Qualification Metric:
To evaluate a building’s qualification status, a metric and a thresholds should be defined. After empirical testing, the recommended metric for CalTRACK 2.0 is CV(RMSE).

Justification:

The properties of the CV(RMSE) metric penalize buildings with outlier energy use. Due to their significant effect on portfolio uncertainty, it is recommended that buildings with outlier energy use are eliminated from portfolios. The CV(RMSE) metric makes removing outlier buildings from portfolios more important.
CV(RMSE) is not sensitive to close to zero individual usage values. This makes it more robust than MAPE and similar metrics.

Building Type:
Building type can be a useful identifier for aggregators to determine a building’s suitability for a portfolio. In figure 1, the relationship between energy consumption and CV(RMSE) is measured by building type. The size of each dot corresponds with the number of meters per building type.

The CalTRACK methods will be most effective for buildings in region A, which have relatively low energy consumption and low CV(RMSE). Buildings in region B are high energy consumers. These buildings often have a single meter tracking consumption for various sub-buildings with mixed uses, which make it difficult to quantify the effect of an energy-efficiency intervention on overall consumption. These buildings will likely require custom M&V and not qualify for CalTRACK. The buildings in region C have high CV(RMSE). The high CV(RMSE) is likely due to correlation in energy usage that is not specified in the model, such as seasonality. These models should not qualify for CalTRACK.

CV(RMSE) Threshold for Building Qualification:
Due to differences in model quality, portfolio size, and building type between aggregator datasets, it is difficult to establish a universally applicable building-level CV(RMSE) cut-off. The graphs below visualize the relationship between building-level CV(RMSE), portfolio uncertainty, and building attrition. The various graphs show these relationships with different building types and portfolio sizes. While analyzing the graphs, consider that procurers and aggregators tend to be more concerned with portfolio-level uncertainty and building attrition than the building-level CV(RMSE), especially for pay-for-performance programs and Non-Wires Alternatives procurements.

The results show that strict building-level thresholds of 25% CV(RMSE) result in low portfolio uncertainty, but significant building attrition. Also, the results vary depending on the portfolio size and building type. From an aggregators perspective, it may be preferable to adopt a less restrictive building-level CV(RMSE) threshold and focus on minimizing building attrition with respect to a strict portfolio uncertainty threshold.

Recommendations:
We are recommending that the specific building eligibility requirements be generally left to the procurer, who can set the requirements that align best with their goals for a procurement, provided these are specified clearly upfront. CalTRACK can provide general guidelines as follows.

For use cases where confidence in portfolio-level performance is required (e.g. aggregator-driven pay-for-performance, non-wires alternatives (NWA) procurements), we recommend using a permissive building-level CVRMSE threshold (100% is recommended), but requiring that a portfolio-level metric be respected (e.g. weighted mean CVRMSE or portfolio fractional savings uncertainty). The portfolio-level threshold will be a policy decision and may differ depending on the use case (e.g. NWA procurement may require less than 15% uncertainty, regular pay-for-performance program may require 25% to align with ASHRAE Guideline 14 etc.)
For use cases where confidence in individual building results is required (e.g. customer-facing performance based incentives), ASHRAE Guideline 14 thresholds may be used.

Other Reading Cited in the Working Group Meeting today:
Normalized Metered Energy Consumption Draft Guidance CPUC
ASHRAE Guideline 14

Homework:

Comments on Recommendations for Building Qualifications
Suggestions and recommendations for hourly models
Revisiting other requirements in terms of hourly models
Tests for hourly models

Next CalTRACK working group meeting will be April 12, 2018.

2 Comments

Building Qualification Method Discussions Continue

3/26/2018

0 Comments

Week Seven CalTRACK Update

A quick update for this week and a reminder of the working group meeting:
Thursday, March 29th at 12:00 (PST) in which we will cover:

Final discussion of building qualification methods
Introductory discussion on Hourly Methods

During week seven of CalTRACK, consideration of building qualification methods continued. This week’s working group meeting will conclude the discussion of building qualification methods and launch the testing period. Comments or test results should be added on GitHub issues early next week to ensure they can be considered before proposals are finalized.

Homework: