Building Qualification Observations and Recommendations:
Main observations that were driving the recommendations:

• Stricter building-level thresholds do not necessarily result in more confidence in portfolio savings.
• Thresholds would vary by data granularity. Distributions of building-level metrics also vary widely by building type, location and climate zone.
• Bias must be minimized for portfolio aggregation to work.
  
  

The building qualification recommendations are dependent on use case:

1. For use cases where confidence in portfolio-level performance is required (e.g. aggregator-driven pay-for-performance, non-wires alternatives (NWA) procurements), we recommend using a permissive building-level CVRMSE threshold (100% is recommended as a default), but requiring that a portfolio-level metric be respected (e.g. portfolio fractional savings uncertainty).

The portfolio-level threshold will be a policy decision and may differ depending on the use case (e.g. NWA procurement may require less than 15% uncertainty, regular pay-for-performance program may require 25% etc.)

1. For use cases where confidence in individual building results is required (e.g. customer-facing performance based incentives), ASHRAE Guideline 14 thresholds may be used.
   
   

Notes:

• The FSU thresholds will be context-specific because the value of certainty is not uniform across projects and procurers. For example, a utility may be willing-to-accept a portfolio with 25% uncertainty in the context of a pay-for-performance program. The same utility may require 15% uncertainty for a non-wires alternative project.

CVRMSE and FSU are different metrics andare described in this paper: (A Comparison of Approaches to Estimating the Time-Aggregated Uncertainty of Savings Estimated from Meter Data)

Goals for Hourly Methods:
Hourly models are necessary for estimating the load impact of energy efficiency. This makes hourly energy savings important information for aggregators and utilities in an energy efficiency marketplace.
Our goal is to establish suitable methods for calculating whole building hourly energy savings for residential and commercial buildings. Additionally, the methods will include guidelines for aggregating site-level savings.
​
Discussion Topics:
We have allotted three weeks to test and discuss hourly methods. Below are some important topics that will need to be addressed:

1. Data Requirements and Sufficiency:
   • Hourly weather data can be unreliable. We will need to establish guidelines for hourly weather data sufficiency and methods for accommodating missing weather values.
   • Data sufficiency requirements for hourly meter data are needed.
   • Hourly data cleaning methods must be defined.
2. Methods:
   • It may be beneficial to adjust our model selection criteria for hourly methods.
   • Aggregating many hourly models will result in high portfolio uncertainty. As a result, it may be necessary to reassess the calculation of portfolio uncertainty in hourly savings.
3. Models:
   • We will need to determine the most suitable hourly model for CalTRACK. Two open source models to begin the discussion are:
     1. The TOWT model (time-of-week and temperature) from Lawrence Berkeley National Lab
     2. ECAM model from SBW consulting​

We look forward to future discussions on GitHub regarding these topics.

Time Of Week and Temperature Models:
To start the discussion of hourly methods, it is helpful to consider to existing open source tools. Lawrence Berkeley National Lab has developed an open source time of the week and temperature (TOWT) model to calculate hourly energy savings and is part of the RMV2.0 - LBNL M&V2.0 Tool. A TOWT model predicts hourly energy savings by utilizing hourly temperature data instead of daily or billing period HDD and CDD.  Another is ECAM (ENERGY CHARTING & METRICS) developed by Bill Koran at SBW consulting.  

We look forward to discussion on the working group call about these approaches, and more to frame the approach to empirical tests for the CalTRACK 2.0 hourly methods.