CalTRACK Methods Development Process

Picture
Click To Enlarge
The CalTRACK working group met biweekly to discuss each of the following six areas in detail and arrive at a consensus on questions that need to be answered, potential solutions, and, ultimately, draft requirements. The summary of the technical working group discussion on each of these issues can be found in the agendas and findings below:

Summary of Meeting Notes and Agendas

Key Issues, Solutions, and ReQuirEments

1. Data Sources and Access

Picture
In order for CalTRACK to work, the system must have access to standard sources of project data, usage data, and weather data. This access must be structured, automated, reliable, and secure.

Key Issue
  • Standard Sources
    • Given that utilities do a lot of pre-processing of monthly billing data, which version of billing data should be used in the CalTRACK System?
    • What are the standard sources of weather data?
    • What resolution of usage data will we use: monthly, daily, less? Monthly makes it easier to eliminate variables.
    • What sources of customer information are there beyond usage data stored by the IOUs?
    • Should CalTRACK attempt to incorporate any event data or customer behavior data?
    • Should CalTRACK require access to a standard comparison group for each IOU or area? If so, how will access and consent be handled if eemeters are stood up outside IOU firewalls.
    • How will CalTRACK gain access to data from municipal utilities?
    • CIS data will be available for the software feedback use case but may not be available for the P4P use case?
  • Authentication
    • If the CalTRACK engine is housed behind the IOUs firewall, how will it consistently gain access to project information
    • How will?
  • Consent
    • Are there issues of customer consent that arise during the pilot phase?
    • Is Share My data Integration necessary for CalTRACK?
  • Data exchange format
    • What format will each of the datasets need to be in for the pilot phase?
    • What format will datasets need to support as part of the ongoing system?
  • CalTRACK Instance Location
    • Where will CalTRACK Instances Live
      • This is critical to determining data access because whether or not a CalTRACK instance is housed at an IOU creates different challenges to data access for each of the main datasets
Potential Solutions
  • Location of CalTRACK Instances    
    • House at IOUs behind firewall on IOU infrastructure
    • House at program implementers
    • House in cloud hosted instances
  • Standard Sources
    • Monthly billing data
      • There was agreement that final billed usage data should be used for monthly billing data analysis.
      • Field Requirements
        • Flags for estimated
          • how hard is this for IOUs?
        • Flag for the presence of net metering
        • (Any other flags useful for segmenting analysis?)
        • Bill start date
        • Bill end date
        • Usage
      • Each IOU will need to identify the data point person(s) for each necessary data input
    • Interval data
      • Because IOUs interval data doesn’t go through the same cleaning procedures as monthly billing data, during the pilot phase the CalTRACK system will use the same version of the interval data that is accessible to customers through Share My Data.
      • Agreed on source: Same source as AMI data
      • Use current IOU imputation procedure (VEE process)
      • Goes through data cleaning process
      • Ensure consistency with daylight savings in use of AMI data
        • standard for timestamp in California is GMT
      • Field Requirements
        • Timestamp
        • Usage
        • Flags for net metering & estimation
    • Weather data
      • Possible Sources
      • It was agreed that for the pilot, CZ2010 data would be used for weather normals. There is still an open question about whether to use the 16 city zones or the complete 86 climate zones, but most of the stakeholders expressed a preference for the 86 climate zones.
      • The CalTRACK team agreed to empirically compare the results from the use of 16 vs 86 climate zones to show the degree to which they differ.
      • What has been used in previous evaluations
        • While Ex-ante has used 16 climate zones
        • For billing analysis, used full set of 80+ NOAA weather stations
        • Work order 46 has comparison using the 2 data sources (Jarred will link to it)
        • Does NOAA subscription offer imputation?
          • We will talk with Jeff Chen about NOAA weather service providing imputed data
          • Needs to be consistent with what is being reported to customers
          • Talk with TYM3 to know if they are doing
          • Talk about collaborating on a rolling 10-year window (CEC will reach out to BlackBox technologies)
    • Project data
      • Project data will initially come from a combination of implementer’s databases as well as the XML files from software vendors.
      • The main discussion around project data focused on the required fields. In particular, the crucial decision about which date fields will be used to set the beginning and end of the intervention. It was agreed to initially use project application date and project completion date because they were likely to be the most consistent across implementers and IOUs
      • There was some discussion about whether there are events in the CIS systems about each account that should be included.
      • How do we ensure availability and consistency of software vendors & contractors in project data?
        • CalTRACK Alpha has required both XML files and database access
      • In evaluations
        • Standard tracking data
          • Project ID (could link to HP-XML)
            • being populated by software in HP-XML file
            • needs to be aligned with standard tracking database so link can be made
          • Service ID (links to billing data)
          • HP-XML File
          • Zip code
            • Define a consistent key
              • Use CZ2010 Zip code map for 80+ weather stations
    • Counterfactual/Comparison Group
      • No solutions were arrived at on the need or mechanics of comparison groups for bias correction as part of the data source discussion but it will be part of the methods call.
      • How do we ensure comparison group selection includes filtering for solar?
      • CIS should be
  • Authentication
    • Accessing project data
      • Program implementers suggested using their current IOU jobs portals to submit programmatic data for the CalTRACK system.
    • Accessing usage data
      • The issue of third-party access to usage data (program implementers) was brought up and it was decided that for the pilot, CalTRACK Instances will be housed by the IOUs and program implementer will share project information back to the IOUs through their existing portals.​
  • Consent
    • No particular solutions with consent were discussed.
  • Data exchange format
    • Usage data
      • Stakeholders discussed whether there was any internal standardization of usage data among IOUs. It was decided that, during the pilot phase, issues of data exchange format for usage data would be resolved by writing custom ETLs for each IOU into CalTRACK, rather than attempt imposing a standard schema.
      • No decision made on data resolution (monthly vs. daily)
    • Project data
      • HP-XML will be the standard data exchange format for project data
Draft Requirements
  • Location of CalTRACK Instances for data access
    • CalTRACK instances used in generating canonical savings estimates will meet utility security requirements to allow for direct usage data access
  • Standard Data Sources
    • Monthly Billing Data
      • Source
        • Final monthly usage data
      • Field Requirements
        • Unique account identifier
          • SAID
          • SPID
        • Bill start date
        • Bill end date
        • Usage
        • Fuel type
        • Flags for estimated usage
        • Flag for the presence of net metering
        • Rate classification
    • AMI Data
      • Source
        • Post VEE data available to customers through share my data
      • Frequency
        • Gas: daily
        • Electric: 15-minute interval data (where available). 60-minute if 15-minute not available
      • Field Requirements
        • Unique account identifier
          • SAID
          • SPID
          • Premise ID (char_prem_id in PG&E’s system)
        • Flags for estimated usage
        • Flag for the presence of net metering
        • Direction of Usage
        • Date and Timestamp
        • Usage
        • Rate classification
    • Weather Data
      • Source
      • Field Requirements
        • Avg Temperature
        • Timestamp
        • Weather station identifier
        • Lat/Long
        • Elevation
        • Imputation flags
    • Project Data
      • Source
        • Software HP-XML Source Files
        • Project Data CRM
      • Field Requirements
        • Project ID
        • Service ID (to be anonymized)
        • Zip Code
        • Project start date
        • Project completion date
        • Installed measures
        • Anonymous contractor ID (Home upgrade use case)
        • HP-XML Base Fields (Home upgrade use case)

2. Data Integration

Picture
One of the most challenging aspects of the CalTRACK system will be ensuring reliable data integration. This means having a standard way of reliably matching usage data collected by utilities with project data collected by contractors and weather data collected by NOAA. Several key issues were identified:

Key Issues
  • Matching across datasets
    • How will accounts be matched between systems?
    • Do HPXML files contain the necessary identifiers to link with customer usage data?
    • Should there be naming conventions for HP-XML files to aid record linkage?
    • Is there any need for fuzzy matching?
  • Deduplication
    • How does CalTRACK deal with the same household appearing multiple times within a program?
    • What should be done in the case of multiple meters?
    • Additional projects in other programs?
  • Unmatched data
    • What should be done with projects that don’t match to one or more required datasets?
  • Weather station matching
    • What rule will be used for matching?
Potential Solutions
  • Matching across datasets Unique identifiers
    • Exact Matching on Unique Identifiers
      • Use standard tracking data
        • Project ID
        • Service ID
        • HP-XML Unique Identifier
    • Fuzzy Matching
      • BIG parses HP-XML files, then match with PG&E
        • HP-XML file is uploaded (does not include service ID)
        • Customer is verified through a fuzzy matching process
          • performed by desktop reviewer at PA
          • Address / customer name = Fuzzy match
        • Feed HP-XML
 
  • Deduplication
    • If a home appears multiple times within a project database, and the project dates are the same the most complete version of that home should be included
    • If a home appears multiple times within a project database and the project dates differ because [add language], the start date of the intervention will be the earliest of the project start dates across projects and the end date for the intervention will be the latest of the project end dates
    • Double dipping incentives across multiple programs on the same ECMs
      • desktop review check
      • PG&E rebate processing check
    • Attribution issues associated with participating in multiple programs
      • checking across all programs is not easily done
      • flag edge cases (large saving) for evaluation
  • Weather station matching
    • Options for matching weather station to project
      • Current CZ2010 zip code to weather station mapping
      • Distance
      • Microclimate
      • Fit (correlation to usage)
    • Issues
      • SAIDs can change often
        • new customers
        • rate increases
        • this gets resolved during the payment process
        • E
      • CalTRACK will need to have the history of SAIDs to be able to get the full history
      • PGE does not have consistent mapping of SAID to building
        • Can we extend the Building ID concept to residential?
      • MacroConsumption Project
        • In 2010-2012
        • Premise ID was put together for aggregate analysis
        • DNV-GL handles residential data for this
        • Has standardized 2012-2014 monthly billing data
        • All of this data will go to UCLA for Energy Atlas project
        • Issues Identified
          • Explicit flags for CARE, FERA, Net Metering
          • If you change to time of use rate, or other program, there is not a flag
          • If there is a change in account holder name, there is no explicit flag for change
          • Share my data doesn’t have the ability to stitch together after  account holder change
          • Share my data provides rate plan ID numbers
      • DNV-GL can push list to IOUs
      • Can the Building ID mapping be made consistently available
        • Process
          • Data request goes out in Feb/March
          • Process data comes out in May
          • DNV-GL will make data request on behalf of IOUs
          • CPUC initiates a secure data transfer request
      • Can we include new building to account mapping
Draft Requirements
  • Matching project data
    • Matching to Project IDs to Service IDs on the implementer side
    • For other third parties that do not have access to SAID, use fuzzy matching procedure for linking project information and usage data
      • Use site address + customer name + phone number + email from project data and account address + customer name +phone number + email from IOU customer information system to link project ID with service ID for consistent link to usage data
      • Record linkage will be performed using dedupe, an open source python package for fuzzy matching.
      • For aggregators, they will need to get SAID from customer
      • [Add guidance on Share My Data needing SAID]
  • Matching weather stations to projects
    • Default to using 86 station standard mapping of zip code to CZ2010 weather files
    • Allow IOUs to substituted their own weather station mappings in their implementations of CalTRACK, as long as they report back their modifications to a statewide CalTRACK mapping dataset so that we end up with a unified statewide zip code to weather station matching dataset
  • Deduplication
    • If a home appears multiple times within a project database, and the project dates are the same the most complete record for that home will be the record used in CalTRACK
    • If a home appears multiple times within a project database and the project dates differ because there are multiple measures installed associated with the same incentive program, the start date of the intervention will be the earliest of the project start dates across projects and the end date for the intervention will be the latest of the project end dates
      • The technical working group will review literature and possibly run ex-post experiments on multi-intervention homes to test fidelity of sequential savings estimates
  • Unmatched data
    • Projects that are unmatched to usage data will be listed in CalTRACK for data integrity reporting, but will not be included in any estimation procedures and will not have estimated savings

3. Data Cleaning and Quality Checks

Picture
In order to provide high-quality, actionable results, the CalTRACK system will need a standard, reproducible set of data cleaning procedures that are consistent with the idiosyncrasies of the standard input data sources and the requirements of the estimation methods applied during analysis. 

Key Issues
  • Missing values
    • What should be done in cases where fields have missing values?
  • Extreme values
    • What rules have been used in the past for dealing with outliers, extreme values?
  • Miscoded values
    • What rules should be applied for screening for and dealing with values that are likely miscoded?
  • Extreme project lengths
  • Insufficient data
    • What rules should be applied to ensure there is sufficient usage data to fit a model and provide reliable results?
  • Imputation & deletion
    • Should there be linear interpolation for missing values?
    • When should observations be interpolated vs. thrown out?
Potential Solutions
  • Resources on cleaning methods
  • Missing values
    • Weather: Missing weather data could be imputed (what was done in CalTEST)
      • Random forest imputation scheme
    • Usage
      • missing values in usage are generally estimated
      • estimated values should be left out of billing aGeenalysis
  • Extreme values
    • Usage
    • Project Data
      • ​Miscoded values
        • Force
        • String matching
  • Insufficient data
    • Usage (Monthly)
      • 12 months pre-retrofit
      • Post retrofit data sufficiency for estimation will be dealt with in post-estimation model fit criterion
      • Total annual savings estimates will require 12 months post-retrofit
    • Usage (AMI)
      • 12 months pre-retrofit
      • Post retrofit data sufficiency for estimation will be dealt with in post-estimation model fit criterion
      • Total annual savings estimates will require 12 months post-retrofit
    • Weather
      • Match to nearest weather station with sufficient data
  • Adjustments (if values change during performance period)
    • Usage
      • Use most up-to-date meter read
    • Project data
      • Use most up to date value
    • Weather data
      • Use
  • Imputation & deletion
    • Estimated reads
    • Single-value linear interpolation may be allowable for hourly weather data
      • CalTEST filled in up to 5 sequential missing values using random forest imputation
      • Share My Data has two flags
        • quality
        • estimated vs actual
      • BPI 2400: combine
Draft Requirements
  • Missing Values & Imputation
    • Weather
      • Hourly
        • Hourly weather data from GSOD will not be imputed
        • Days with more than 5 contiguous hours of missing data will be thrown out
      • Daily
        • GSOD daily averages will not be imputed
        • Months with more than than 3 missing days will be thrown out
    • Usage
      • Missing values where the cumulative value is in the following period, the cumulative number of days between the two periods will be used to generate the UPD for that period
      • Missing usage values with no cumulative amount in the following period will be counted against data sufficiency requirements
      • Homes with Net Metering will be dropped from the analysis
  • Estimated values & deletion
    • Estimated usage data will be used for estimation, but estimation flags will be added to the post-estimation
    • Estimated data will enter the same way missing cumulative values do (see above)
  • Extreme values
    • Usage
      • Negative values or values with reverse direction of flow will be treated as missing and count against sufficiency criterion. The account will also be flag within CalTRACK for possible net metering if it does not currently contain a net metering flag
      • It is assumed at all IOUs comply with DASMMD.
      • AMI data will have the DASMMD pass/fail criterion rerun, with failing values coded as missing. ((highest peak - third highest peak)/third highest peak) <= 1.8
    • Project Data
      • Extreme project lengths (gap between project start date and project end date longer than 3 months) will be treated as true and impact estimation only through data sufficiency requirements.
      • Files without project start dates are thrown out
  • Sum Check
    • If both monthly and AMI data are available for a home, CalTRACK will run a sumcheck and use the DASMMD criterion for pass/fail. If it fails, the home is flagged and treated as having missing usage data so no estimation is run on it.
  • Miscoded values
    • Miscoded strings in project data will be deduplicated and matched (fuzzily) to closest value
    • Miscoded dates
      • Implausible day values (>31) will be coded as the beginning of month if project start date and end of month if project end date so that the entire month included in the intervention window
      • Implausible month and year values will be flagged and that home not included in estimation.
  • Data sufficiency
    • Usage (Monthly)
      • 12 complete months pre-retrofit for monthly billing data to qualify for estimation or 24 months with up to 2 missing values from different, non-contiguous months
      • Post retrofit data sufficiency for estimation will be dealt with in post-estimation model fit criterion
      • Total annual savings estimates will require 12 months post-retrofit
      • Do we include homes that have changed tenants
    • Usage (AMI)
      • 12 months pre-retrofit
      • Post retrofit data sufficiency for estimation will be dealt with in post-estimation model fit criterion
      • Total annual savings estimates will require 12 months post-retrofit
    • Weather
      • There should not be problems with data sufficiency for weather
  • Project or Home Characteristics
    • Exclude homes with PV for whom production data is not available
  • Value Adjustments (if values change during performance period)
    • Usage
      • Use most up-to-date meter read for
      • Log prior values and prior estimates
    • Project data
      • Use most up to date values for estimation
    • Weather data
      • Use most recent daily weather value for estimation

4. Data Analysis

Picture
Agreement on the standard methods for calculating savings from monthly and hourly usage data is one of the most important for the technical working group to agree on. While this will build on and attempt to harmonize with national efforts for standardization of automated M&V, the working group may face California-specific issues that require tailoring standards to the regional use case. 

Key Issues
  • What’s old, what’s new
    • California Evaluation Framework
      • “Data source and analysis methods: The suite of tools available to conduct impact evaluations will need to evolve as the research questions change. Traditional engineering analysis, building energy simulations, metering and monitoring protocols, and statistical analysis of billing and survey data may need to be combined in new ways to answer emerging research questions. The application of these tools will need to be described in a direct and straightforward manner to meet the needs of a diverse group of program planners and implementers. In some cases, new research questions may call for new analysis tools. Data standardization efforts should be used whenever possible to gain the maximum usefulness of any data collection effort.“
    • Software choice and the consistency vs improvement tradeoff
    • Scalability and the precision vs participation tradeoff
    • Incentives and the cost vs additivity tradeoff
  • Gross savings estimate using monthly billing data
    • Basic billing analysis is pretty settled. Is there any reason to poke the bear?
    • Comparison groups for gross: cost-benefit analysis
  • Gross savings estimates using AMI methods and hourly counterfactual generation
    • Which of the much richer class of emerging methods for AMI data analysis?
    1. What should be the selection criteria for AMI methods given lack of industry consensus?
    2. What does the literature currently say about precision
    3. Should CalTRACK support methods for sequential savings estimation?
  • Bias adjustment and comparison groups
    • Should there be bias adjustments on gross savings to control for exogenous factors that differentially impact treatment?
    • Should CalTRACK support quasi-experimental methods be used for bias adjustment
    • Should CalTRACK take advantage of any experimental program design (RCT,REC) that is taking place in IOUs?
    • Is it possible to estimate bounds on bias terms that are tolerable for CalTRACK purposes.
    • Ken: comparison groups do very little to get you to net
  • Defining pre-post period, especially with ongoing interventions
  • Do we include homes that have changed ownership?
  • Model selection
    • What should the fitness criteria be for balance point temp selection? (parameter search, or climate zone defaults)
  • Standard errors and confidence intervals
    • How should errors propagate given independence assumptions may not hold
  • Post-estimation sufficiency and fitness criteria
    • What are the fitness criteria for inclusion in the savings estimate pool?
  • Extreme value considerations
    • Should fitness criteria take into account outlier prevalence (square vs absolute measures of deviation)
  • Model validation
    • Should there be ongoing out-of-sample testing, especially for AMI methods, for ongoing model validation?
    • What are current approaches to ongoing model validation for automated M&V?
Potential Solutions
  • Gross Savings using Monthly Billing Data
    • Standard Practice
    • Gross savings estimates for each individual site
      • Two main estimator choices
        • Two stage
          • Benefits
            • Standard practice
            • Gives you a site-level estimate
            • More flexibility in how you want to aggregate
            • EUC is based on site-based savings
          • Costs
        • Pooled
          • Benefits
            • Long history for program evaluation
          • Costs
            • Need to determine pool one the fly and recalculate for each block group
        • Working group consensus
          • CalTRACK should use two-stage model
      • Two-stage Model Details
        • First Stage Equation
        • Choosing and
          • Options
            • Static by geography across all homes
            • Per-site optimal search with regional defaults
              • Objective function
              • Need a strict step size or fitness criteria for opting back to default
          • Discussion with EPA about regional publishing of set points and balance points from smart thermostat project
          • Issues
            • This optimization can be dominated by noise and that can lead to chases into
        • Comparison group choice and factor adjustments on gross
          • Goals
            • Methodologically Sound
            • Replicable
            • Transparent
          • Use Cases for CalTRACK and uses of Comparisons Groups
            • P4P Pilot
            • Software comparisons
              • What is “good enough” (i.e. CalTest cutoffs)
          • Reasons to use comparison group for site-level estimates
            • Evidence that there is residual bias after 12 months post-treatment
            • DNV-GL 2010-2012 Report
              • Savings for every IOU increased with use of comparison group
              • No within/without comparison for 2013
            • Comparison group can actually increase savings (direction of population-level trends and effects of global exogenous factors can be in either direction)
            • Important to hold as many things constant as possible
            • Do we want pay aggregators for factors that are out of their control
            • Nonparametrics help address nonlinearities
            • What are those factors
              • Non-temperature weather changes
              • Macroeconomics effects
              • Demographic trends
              • Energy pricing
              • Market effect (efficiency over time)
          • Comparison group options
            • Local residential building population aggregate use statistics based on public data
              • Examples
              • Pros
                • Openly available
                • Reduces uncertainty for program administrators and third party participants
              • Cons
                • Will attenuate estimates (treatement included in total sample), so estimate between gross and net
                • Relies on small relative proportion assumption
                • Time lag for updates
                • Sustainability
                • Not able to match on usage, only on geography. Matching to the conditional mean only conditioning
                • Probably last resort
            • Non-participant local public usage aggregations published by utilities
              • Examples
                • None to date, IOUs would have to create as part of CalTRACK
              • Pros
                • Could be made public, lower uncertainty
                • Could be made up-to-date (published monthly)
                • Reduces inclusion bias and proportional concerns
              • Cons
                • Non-matched sample (or week conditioning on match) would likely attenuate estimates, leads to estimate between gross and net
            • Non-participant matched sample
              • Examples
                • Typically done in two-stage models using propensity score matching
                • Train a nonparticipant model and publish segmented adjustments publically
              • Pros
                • Standard practice
                • Better comparison group than prior two options
              • Cons
                • Cannot be made public, increases uncertainty and forecasting
                • Estimates between gross and net
                • There may be issues related to data access and publication
                • May not be able to use third-party data for segmentation
              • Discussion
                • How feasible
            • Prior and future participants
              • Pros
                • May be best sample match
              • Cons
                • Second-year participants won’t have new entrants
                • If participants are changing rapidly under
              • Discussion
                • We will need to view stability of participant composition​
      • Post-estimation
        • Fitness criteria
          • Minimum fitness criteria for inclusion in portfolio estimate
            • CVRMSE
            • R^2
              • Minimum .2 for electric
              • Minimum .5 for gas
        • Standard NAC calculation using CZ2010
          • Use in realization rate calculation
        • Estimates to actuals comparisons using current year data
          • Useful for continuous gross savings estimates at portfolio level
          • Higher SE
  • Gross Savings using AMI (15-minute, hourly, or daily) Data
    • Standard Practice
    • Model selection
      • Rich set of methods to choose from
        • Typical Regression Approaches
          • Basic two-stage regression with use rolled up to daily HDD/CDD (mirroring monthly billing methods)
          • Time of week + temperature models
          • Bin modeling
        • Time Series and Signal Processing Approaches
        • Machine Learning Approaches
          • Tree-based Methods
            • Random forest
          • Ridge Regression
          • Lasso Regression
          • KNN
        • Ensemble methods and GLS
        • Gaussian Mixtures
        • Bayesian Approaches
          • Hierarchical Models
          • Structural Time-series Models and more advanced counterfactual generation
        • Deep Learning
          • Recurrent Neural Nets
      • White paper on model selection
      • Common dataset needed for evaluating over model choices for CA
        • Options
          • Out of sample testing on pre- period of participating home
            • Used by Catherine Wolfram in Schools paper
          • Verified non-intervention control group data used for out of sample testing
            • Used by Granderson et al.
        • Suggestion
          • Develop out of sample testing on pre-retrofit period (3 month censoring)
          • Out of sample testing on future participants (year two) in year one
      • Test Metrics
        • Options
          • Normalized Mean Bias Error
          • Coefficient of Variation of the Root Mean Squared Error
          • Total bias
          • Total error
          • Mean bias
          • Mean absolute percent error
          • Normalized mean bias error
          • Root mean squared error
          • Coefficient of determination
        • Prior technical working groups in commercial selected NMBE and CVRMSE. Any reason not to?
          • Proposed Approach
            • Portfolio wide
            • Subset by customer segments
            • Subset by time period
            • Weighted CVRMSE that puts more weight on peaks
      • Model uncertainty
        • See white paper
      • Starting with the best out of sample performance
        • Ensemble methods seem to be winning
          • Extremely randomized trees with mean week
    • Choosing and
      • Cannot use optimal regression approaches for most more advanced baselining approaches
Draft Requirements
  • Site-level Gross Savings using Monthly Billing Data (both electricity and gas)
    • Two-stage estimation following UMP, California Evaluation Project
    • First Stage Equation
    • Choosing and
      • Per-site optimal search with regional defaults
        • Objective function
    • Comparison group adjustment using future participants will be tested
    • Second Stage Estimates:
      • Cumulative gross savings over performance period
      • Annualized gross savings in a normal year
      • Annualized gross savings in current year
  • Site-level Gross Savings using Daily usage totals rolled up from AMI data (both electricity and gas)
    • Two stage estimation with additional fixed effects for day of week and holidays
    • Comparison group adjustment using future participants will be tested
    • Second Stage Estimates:
      • Cumulative gross savings over performance period
      • Annualized gross savings in a normal year
      • Annualized gross savings in current year
  • Site-level gross savings estimation using hourly electric data
    • Evaluation of more advanced baselining methods will use the following testing procedure
      • A test set of prior EUC participant homes with at least 24 months prior use will be selected
      • Traces will be split between baseline and reporting periods based on project start date
        • Only baseline periods will be use for testing
      • Baseline period will be split into first and second year
      • Candidate models will be fit on year-one traces, including the monthly and daily models outlined above (for model comparisons)
      • Year-two predictions will be fit based on year-one parameter estimates
      • The CVRMSE will be calculated using the second year trace and prediction for all test homes
      • The average CVRMSE will be calculated over the sample of test homes as the relevant test statistic for that method
      • CVRMSE distributions for all test methods will be compared by beta testers
      • The best performing will be replicated by beta testers
      • If there is significant improvement over the standard UPD model, for the best performing model while also being sufficiently computable, interpretable, and having meaningful estimates of uncertainty, beta testers may recommend a new AMI method for CalTRACK.​

5. Data Aggregation

Picture
Because the goal of CalTRACK is to provide feedback on the performance of portfolios of homes to a variety of stakeholders, there are a set of critical technical issues around how results from individual homes are aggregated and reported. While the working group did not have a chance to discuss these issues at length, the following issues were presented to the group for their consideration:

Key Issues
  • Portfolio savings totals, averages, & confidence intervals
  • Aggregation
    • Do we annualized before or after aggregation?
      • Savvy annualizes after because you many want to upweight projects with good fit and downweight bad fit
  • Minimum aggregation rules
  • Savings attribution & decomposition
  • Anonymization
  • Secure exchange and provenance
  • Deduplication & entity resolution
Potential Solutions
  • Portfolio-level savings statistics from monthly billing analysis
    • Portfolio total annualized gross savings
      • Sum annualized gross savings over all qualifying projects
      • Inverse-variance weighted average?
      • Relative precision at 90% confidence
    • Portfolio total cumulative gross savings
      • Sum cumulative gross savings over all qualifying projects
      • Relative precision at 90% confidence
    • Portfolio average cumulative gross savings over reporting period
      • Mean gross savings over all qualifying projects
      • Relative precision at 90% confidence
    • Portfolio average annualized gross savings over reporting period
      • Mean gross savings over all qualifying projects
      • Relative precision at 90% confidence
    • Portfolio rolling monthly gross savings
    • Portfolio realization rate over reporting period
      • Mean of annualized gross savings at the site level over predicted savings at the site level for entire portfolio
      • Relative precision at 90% confidence
  • Population-level savings statistics
    • Total annualized gross savings
    • Average annualized gross savings
    • Average cumulative gross savings
  • Considerations on estimating uncertainty
    • Ensure clustered standards errors
    • Relative precision at 90% is a requirement
    • Use empirical estimates of the variance
    • Use pooled and compare to two stage approach
Draft Requirements
  • Aggregate Metrics for CalTRACK
    • Portfolio-level metrics ​

6. Data Reporting

Picture
Ensuring CalTRACK results are accessible to a broad set of stakeholders will require important stakeholder engagement in defining the views and reports that CalTRACK products. The CalTRACK system must allow for model outputs to be accessed and viewed in a flexible, customizable, and authorized way. This will include defining a standard API schema that can allow outputs to integrate back into stakeholder’s enterprise systems and run reports, as well as have various dashboards that can be accessed by different stakeholders. While stakeholder engagement, reports, and views were only briefly discussed during the kickoff meeting, some issues that were presented to the group included:

Key Issues
  • Identifying key stakeholders & their information needs
    • How do we ensure that the information being provided to stakeholders that were not present in the kickoff meeting (e.g. contractors) meets their informational needs and is correctly informing their actions?
  • Flexible API for development and innovation
  • Tiered access for user types
  • Ensure outputs can be linked with IOU CIS systems for additional segmentation of results
  • Who are the main users?
    • Home Upgrade Use case
      • Utility
        • Calculate realization rate
        • How contractors vary in realization rate
        • How load shapes are affected
        • How is savings geographically distributed
      • Software Vendor
        • Diagnostic tool for accuracy
        • Realization rate by
          • Measure
          • Climate zone
          • Individual jobs?
      • Contractors
        • Be able to investigate specific jobs
          • May need individual permission for use in
      • Evaluator
        • Rolling annual evaluations
          • CalTRACK sets up standard set of data for the evaluation purposes
        • Benchmark for annualized savings estimates
          • Allowing for comparison between rolling estimates and later
    • P4P Use Case
      • Aggregator
        • What ECMs, contractors, and buildings are creating what savings
        • Load shape (savings in time) attributions
        • Targeted quality assurance
        • Marketing and communication
        • May not need to come out of central CalTRACK
        • These data services are likely too expensive for Utility to provide