Metrics & Model Selection for Building Qualification

Week Five CalTRACK Update

We look forward to another working group meeting Thursday March 15th.  Here are some updates from the activity last week.  

 Building Qualification Summary:
During the past week, we started discussions to establish an empirical method to determine a building’s suitability for CalTRACK methods. Listed below are two topics related to building qualifications that require testing:

1. Preferred Metric: The two main metrics being discussed to evaluate a building’s suitability for CalTRACK methods are Coefficient of Variance Root-Mean Standard Error (CVRMSE) and Mean Absolute Percent Error (MAPE). The ideal metric will effectively determine a building’s suitability for CalTRACK methods, while also being convenient for other stakeholders  in pay-for-performance programs.
2. Model Selection: Due to the tendency of buildings that are unqualified for CalTRACK methods to be intercept-only models, it is proposed that the quality of intercept-only models is evaluated based on the variance of mean usage for daily or billing period data instead of the R-Squared criteria.

Some additional areas that require further analysis before our discussion of building qualifications closes:

1. How do our metrics and methods align with pay-for-performance programs? Do they reflect performance risk and uncertainty?
2. What should we do with disqualified buildings? What is the best way to accommodate them in pay-for-performance programs?

The discussion of building qualifications is scheduled to close on March 14th, so remember to add proposals, comments, and test results as soon as possible.
Homework for Participants:

1. Attend the standing, bi-weekly meeting on Thursday, March 15th at 12:00 PST
2. Review Github and provide feedback on any relevant issues. This will be the last week for suggesting building qualification test plans.