Blog Posts

Time of Week and Temperature Model CalTRACK Application for Site Specific Hourly Savings

5/14/2018

Week Fourteen & Fifteen CalTRACK Update

Review of hourly method proposals continued in week fourteen of CalTRACK 2.0 and will be finalized at the 5/24 working group meeting. Lawrence Berkeley National Lab’s Time-of-Week Temperature (TOWT) model is the specification to be used in CalTRACK 2.0. Mathieu et al. describe the application of TOWT models in Quantifying Changes in Building Electricity Use, with Application to Demand Response.

Overview of TOWT models:
As energy efficiency finds its legs as a grid resource, time dependent savings will be essential to the value proposition. Pay-for-performance programs can leverage this value with accurate building-level energy savings calculations at granular time intervals. TOWT models are one method for calculating energy savings at the hourly level.

Strengths:

Aggregated portfolios for pay-for-performance programs may have high variability in hourly energy consumption due to its nature. This variation is evident in figure 1, which shows load patterns over time for an office building, furniture store, and a bakery. The TOWT model addresses this variability in the following manner:
An “occupancy” proxy is determined using a linear regression model and allows for the hourly data to be segmented based on a building’s occupancy status.
Several “time-of-week” independent variables (one for each hour of the week) are included in the main linear regression model to capture hourly load variation. For example, a restaurant may regularly consume more energy on Friday nights because the restaurant has more customers on Friday nights. This type of variation will be controlled for by the “time-of-week” covariate.
The temperature covariate uses 7 bins of fixed temperature ranges instead of employing a grid search to find the balance points. Due to higher amounts of data in hourly methods, the fixed temperature ranges provide a simpler solution without significant drawbacks.

Weaknesses:

By nature, calculating hourly energy savings requires more granular data. This can make data sufficiency problematic.
Similar to daily methods, energy consumption on weekends or holidays may be different than typical days.
There is autocorrelation in the errors of parameter estimates, which complicates uncertainty calculations.

Homework:

Review draft of billing and daily methods write-up
Review proposals for hourly guidelines on GitHub
The next working group meeting is on Thursday, 5/24

0 Comments

Considering Hourly Methods: Data & Use Cases

5/7/2018

0 Comments

Week Thirteen Update for CalTRACK

During week thirteen, the CalTRACK working group discussed proposals for hourly methods in the standing meeting. The discussions included helpful suggestions of other reference materials as well as variations that may be appropriate for different applications of hourly methods and suggested improvements in CalTRACK 2.0’s documentation. The video from May 3, 2018 is provided at the end of this post.

Hourly Methods
In the development of hourly methods, the goal is to establish guidelines that mirror the methodology in billing period and daily methods. However, hourly methods have unique complexities that require departures from billing period and daily methods. These complexities are identified and discussed below:

Data Management
When compared to hourly data, daily and billing period savings calculations have higher data sufficiency requirements because hourly data contains more information per time period. This characteristic of hourly data supports the two adjustments to hourly data sufficiency requirements listed below:

 1. Usage data sufficiency will be specified in terms of data coverage (or common support) instead of a minimum time period:
Daily and billing period data sufficiency requirements impose a minimum quantity of time observed from a year of data. In hourly methods, usage data sufficiency will be specified in terms of data coverage in the independent variables. In Time of Week and Temperature (TOWT) models, the independent variables are temperature and occupancy. Data sufficiency requirements will be based on LBNL recommendations for data coverage.

2. Missing Data
Temperature has less variation between hours than days or billing periods. Smaller temperature variation between hours increases the likelihood that interpolated temperature values are accurate. For this reason, interpolated temperature values will be allowed in the reporting period for hourly methods. The threshold of allowable interpolated hours will be determined through empirical testing.

Recommendation for Pay-for-Performance Use Case:

In the baseline period, it is recommended to drop hours with missing temperature and usage data.
In the reporting period, a maximum of 6 missing values per day can be interpolated while maintaining minimum data sufficiency requirements.

Time Of Week Temperature (TOWT) Modeling Approach
The TOWT model, originally by Lawrence Berkeley Lab, contains two covariates:
Occupancy:
Occupancy is an indicator variable that takes the value of 1 if the building is occupied in the hour and 0 otherwise. In LBNL’s model, occupancy of a building is defined by:

Using ordinary least squares regression to establish a regression model for a building.
Grouping all observations at each hour. If 65% of the observations for an hour are above the established regression line, that hour is designated as occupied.
If the condition in (2) is not met, then the hour is defined as unoccupied

Temperature:
The TOWT model allows user-defined temperature bins for modeling a building’s weather dependence. We are recommending setting 7 fixed bins with endpoints at 30, 45, 55, 65, 75, 90, in order to cover a wide variety of climate conditions.

Use Case and Uncertainty

Time-aggregated Uncertainty:
In the program evaluation use case, an analyst may be interested in obtaining time-aggregated savings and uncertainty. Due to residual autocorrelation at the hourly level, aggregating hourly uncertainty for larger time intervals creates imprecise standard errors and uncertainty calculations. Instead, we recommend using daily methods with improved ASHRAE or Ordinary Least Squares (OLS) formulations of Fractional Savings Uncertainty (see Koran 2017) for aggregating uncertainty over time periods.

Hour-level Uncertainty Estimates:
For the procurement and pay-for-performance use cases, regression analysis is an effective tool for acquiring point estimates of savings and uncertainty at each hour. If each building is assumed to have independent errors, the uncertainty at each hour for all buildings in the portfolio can be aggregated without an autocorrelation problem.

Methods Documentation
Currently, the documentation for CalTRACK 2.0 is being updated. The first half of CalTRACK 2.0’s documentation will be posted on GitHub to allow the working group to review and comment on the changes in documentation.

Similar to the methods, the development of effective documentation is an iterative process. The documentation for CalTRACK 2.0 will improve by dividing into three distinct documents:

1. Methods
This document outlines the methodology for quantifying billing period, daily, and hourly energy savings while maintaining CalTRACK-compliancy. In CalTRACK 2.0, the Methods will be organized with a numbering system that corresponds to the Methodological Appendix. This will make referencing and accessing the appendix easier.

2. Methodological Appendix
The Methodological Appendix summarizes discussions and empirical testing that justify methodological decisions. The Methods will reference sections in the Methodological Appendix for readers to easily access empirical support for methodological decisions.

3. Field Guide
A document with minimum requirements for an implementation to maintain CalTRACK-compliancy. This is designed to be a practical and accessible checklist for analysts and other implementers of CalTRACK.

Ideas for Future CalTRACK Work
A sandbox has been added to the GitHub site to document proposals for participants to add ideas for future CalTRACK iterations. If you have an idea for CalTRACK 3.0, or beyond that cannot be addressed this year, please add it the the sandbox.

Additional Hourly Methods Resources

LBNL R Code on Time of Week and Temperature (TOWT)
2002 ASHRAE Guideline 14: Section 5 is relevant for data sufficiency requirements.
Uniform Methods Project – US Department of Energy
1. Peak Demand and Time-Differentiated Energy Savings Cross-Cutting Protocol
2. Whole-Building Retrofit with Consumption Data Analysis Evaluation Protocol

Homework

Review draft of billing and daily methods write-up
Review proposals for hourly guidelines on GitHub
The next working meeting is after 3 weeks on 5/23

0 Comments

Hourly Methods Discussions Continue

4/30/2018

0 Comments

Week Twelve CalTRACK Update

During week twelve, we continued our discussion of hourly methods. In the upcoming week, we will analyze test results for hourly methods on GitHub. We will also be talking about how to log ideas for future improvements. The standing meeting to discuss hourly methods will be on Thursday, May 3rd at 12:00 (PST).

Homework:

Use Hourly Method Tools on test data
Report findings on Github
Offer test criteria for hourly models
Attend standing meeting on Thursday, May 3rd at 12:00 (PST)

0 Comments

Hourly Methods Approach & Testing Considerations

4/23/2018

1 Comment

Week Eleven CalTRACK Update

Week eleven was the first week of hourly methods discussion. Developing hourly methods will require discussion and empirical testing of topics unique to hourly methods before we can make final specifications.
Topics that must be addressed include:

Data sufficiency requirements
Modeling approach
Model selection criteria
The effect of aggregating hourly models on portfolio uncertainty

Time Of Week Temperature Models (TOWT):
A proposed model for hourly methods is the TOWT model from Lawrence Berkeley National Labs (LBNL) is shown below.

Notes:

The number of temperature ranges and the balance points will need to be defined. The LBNL model had 5 ranges ( < 55F, 55-65 F, 65-75 F, 75-90 F, 90 F <).
The methods for defining if a building is occupied or not is explained in detail in Phil Price’s Everything I Know About Building Energy Modeling, But Never Told Anyone Before (18:30-30:00).

Homework:

Use Hourly Method Tools on test data
Report findings on Github
Offer test criteria for hourly models

1 Comment

Building Qualifications Results & Recommendations

4/17/2018

0 Comments

Week Ten CalTRACK Update

The CalTRACK working group finalized discussions on Building Qualifications during the first half of Thursday’s (4/12) meeting and dove into the hourly methods during the second half. Bill Koran from SBW consulting provided a helpful overview of hourly models and the ECAM energy data analysis tool. The major findings of this meeting are summarized below:

Video of 4/12/2018 Meeting

Building Qualification Observations and Recommendations:
Main observations that were driving the recommendations:

Stricter building-level thresholds do not necessarily result in more confidence in portfolio savings.
Thresholds would vary by data granularity. Distributions of building-level metrics also vary widely by building type, location and climate zone.
Bias must be minimized for portfolio aggregation to work.

The building qualification recommendations are dependent on use case:

For use cases where confidence in portfolio-level performance is required (e.g. aggregator-driven pay-for-performance, non-wires alternatives (NWA) procurements), we recommend using a permissive building-level CVRMSE threshold (100% is recommended as a default), but requiring that a portfolio-level metric be respected (e.g. portfolio fractional savings uncertainty).

The portfolio-level threshold will be a policy decision and may differ depending on the use case (e.g. NWA procurement may require less than 15% uncertainty, regular pay-for-performance program may require 25% etc.)

For use cases where confidence in individual building results is required (e.g. customer-facing performance based incentives), ASHRAE Guideline 14 thresholds may be used.

Notes:

The FSU thresholds will be context-specific because the value of certainty is not uniform across projects and procurers. For example, a utility may be willing-to-accept a portfolio with 25% uncertainty in the context of a pay-for-performance program. The same utility may require 15% uncertainty for a non-wires alternative project.

CVRMSE and FSU are different metrics andare described in this paper: (A Comparison of Approaches to Estimating the Time-Aggregated Uncertainty of Savings Estimated from Meter Data)

Goals for Hourly Methods:
Hourly models are necessary for estimating the load impact of energy efficiency. This makes hourly energy savings important information for aggregators and utilities in an energy efficiency marketplace.
Our goal is to establish suitable methods for calculating whole building hourly energy savings for residential and commercial buildings. Additionally, the methods will include guidelines for aggregating site-level savings.

Discussion Topics:
We have allotted three weeks to test and discuss hourly methods. Below are some important topics that will need to be addressed:

Data Requirements and Sufficiency:
- Hourly weather data can be unreliable. We will need to establish guidelines for hourly weather data sufficiency and methods for accommodating missing weather values.
- Data sufficiency requirements for hourly meter data are needed.
- Hourly data cleaning methods must be defined.
Methods:
- It may be beneficial to adjust our model selection criteria for hourly methods.
- Aggregating many hourly models will result in high portfolio uncertainty. As a result, it may be necessary to reassess the calculation of portfolio uncertainty in hourly savings.
Models:
- We will need to determine the most suitable hourly model for CalTRACK. Two open source models to begin the discussion are:
  1. The TOWT model (time-of-week and temperature) from Lawrence Berkeley National Lab
  2. ECAM model from SBW consulting

We look forward to future discussions on GitHub regarding these topics.

Homework:

Use Hourly Method Tools on test data
Report findings on Github
Offer test criteria for hourly models
Watch Phil Price Video about energy modeling and hourly methods: Everything I Know About Building Energy Modeling, But Never Told Anyone Before

0 Comments

Building Qualification Criteria Discussions

4/9/2018

0 Comments

Week Nine CalTRACK Update

During week nine of CalTRACK, there were continued discussions on building qualification criteria and some introductory comments on hourly methods. The working group meeting will be held on Thursday, April 12th at 12:00 (PST). We will discuss:

Final comments on building qualification
Proposals for hourly methods testing

Hourly Methods Overview:
Hourly methods are a new addition in CalTRACK 2.0 and were not in the first version of CalTRACK. This task involves testing various hourly modeling methods and recommending a standardized approach to hourly modeling, which can reveal the time value of energy efficiency.
Importance of Hourly Methods:

Hourly methods are a more granular time interval than daily or billing period methods. These granular time intervals are valuable for determining the temporal value of energy savings, by associating savings with grid demand and energy price.

Time Of Week and Temperature Models:
To start the discussion of hourly methods, it is helpful to consider to existing open source tools. Lawrence Berkeley National Lab has developed an open source time of the week and temperature (TOWT) model to calculate hourly energy savings and is part of the RMV2.0 - LBNL M&V2.0 Tool. A TOWT model predicts hourly energy savings by utilizing hourly temperature data instead of daily or billing period HDD and CDD. Another is ECAM (ENERGY CHARTING & METRICS) developed by Bill Koran at SBW consulting.

We look forward to discussion on the working group call about these approaches, and more to frame the approach to empirical tests for the CalTRACK 2.0 hourly methods.

Final Note:
It is important to remember that CalTRACK methods development is an iterative process. The finalized methods for CalTRACK 2.0, just like v. 1.0, will benefit from field deployment and may need to be revised in future years. We expect this process to result in improved and refined methods with successive iterations.

Homework:

Review final results for building qualifications and provide final edits
Review existing tools, practice and concepts for hourly methods
Discuss and provide suggestions for hourly methods on GitHub
Attend the bi-weekly meeting on Thursday April 12th at 12:00 (PST)

0 Comments

Building Qualifications Test Reveals Wide Applicability of CalTRACK Method for Portfolio Analysis

3/29/2018

1 Comment

Week Eight CalTRACK Update

Today we had an exciting working group meeting focused on Building Qualifications with test results and recommendations. We will be stepping into hourly methods in the upcoming week.

3/29 CalTRACK Working Group Meeting Recording

Daily and Billing Period Methods Specifications:
The final comments on daily and billing period methods have been received. The new, finalized specifications are currently being updated you can track the final on GitHub Issue #82. Participants can review the final specifications and track any issues that fed into the final specification prior to their publication at this site.

Building Qualifications:
The CalTRACK methods are equipped to normalize for weather’s effect on a building’s energy consumption. The methods become unreliable when a building’s energy consumption patterns are instead correlated with non-weather variables. For example, irrigation pumps are used on an agricultural cycle rather than in response to temperature changes. It is reasonable to expect higher energy consumption during the growing season and an analyst may suggest re-specifying the model to control for these seasonal variations. Without modification, existing billing and daily CalTRACK models cannot accommodate these nuanced cases. For this reason, buildings that are not well-specified by the CalTRACK model should be identified and removed from portfolios because they have significant effects on portfolio uncertainty. This issue can be tracked on GitHub Issue #71.

Building Qualification Metric:
To evaluate a building’s qualification status, a metric and a thresholds should be defined. After empirical testing, the recommended metric for CalTRACK 2.0 is CV(RMSE).

Justification:

The properties of the CV(RMSE) metric penalize buildings with outlier energy use. Due to their significant effect on portfolio uncertainty, it is recommended that buildings with outlier energy use are eliminated from portfolios. The CV(RMSE) metric makes removing outlier buildings from portfolios more important.
CV(RMSE) is not sensitive to close to zero individual usage values. This makes it more robust than MAPE and similar metrics.

Building Type:
Building type can be a useful identifier for aggregators to determine a building’s suitability for a portfolio. In figure 1, the relationship between energy consumption and CV(RMSE) is measured by building type. The size of each dot corresponds with the number of meters per building type.

The CalTRACK methods will be most effective for buildings in region A, which have relatively low energy consumption and low CV(RMSE). Buildings in region B are high energy consumers. These buildings often have a single meter tracking consumption for various sub-buildings with mixed uses, which make it difficult to quantify the effect of an energy-efficiency intervention on overall consumption. These buildings will likely require custom M&V and not qualify for CalTRACK. The buildings in region C have high CV(RMSE). The high CV(RMSE) is likely due to correlation in energy usage that is not specified in the model, such as seasonality. These models should not qualify for CalTRACK.

CV(RMSE) Threshold for Building Qualification:
Due to differences in model quality, portfolio size, and building type between aggregator datasets, it is difficult to establish a universally applicable building-level CV(RMSE) cut-off. The graphs below visualize the relationship between building-level CV(RMSE), portfolio uncertainty, and building attrition. The various graphs show these relationships with different building types and portfolio sizes. While analyzing the graphs, consider that procurers and aggregators tend to be more concerned with portfolio-level uncertainty and building attrition than the building-level CV(RMSE), especially for pay-for-performance programs and Non-Wires Alternatives procurements.

The results show that strict building-level thresholds of 25% CV(RMSE) result in low portfolio uncertainty, but significant building attrition. Also, the results vary depending on the portfolio size and building type. From an aggregators perspective, it may be preferable to adopt a less restrictive building-level CV(RMSE) threshold and focus on minimizing building attrition with respect to a strict portfolio uncertainty threshold.

Recommendations:
We are recommending that the specific building eligibility requirements be generally left to the procurer, who can set the requirements that align best with their goals for a procurement, provided these are specified clearly upfront. CalTRACK can provide general guidelines as follows.

For use cases where confidence in portfolio-level performance is required (e.g. aggregator-driven pay-for-performance, non-wires alternatives (NWA) procurements), we recommend using a permissive building-level CVRMSE threshold (100% is recommended), but requiring that a portfolio-level metric be respected (e.g. weighted mean CVRMSE or portfolio fractional savings uncertainty). The portfolio-level threshold will be a policy decision and may differ depending on the use case (e.g. NWA procurement may require less than 15% uncertainty, regular pay-for-performance program may require 25% to align with ASHRAE Guideline 14 etc.)
For use cases where confidence in individual building results is required (e.g. customer-facing performance based incentives), ASHRAE Guideline 14 thresholds may be used.

Other Reading Cited in the Working Group Meeting today:
Normalized Metered Energy Consumption Draft Guidance CPUC
ASHRAE Guideline 14

Homework:

Comments on Recommendations for Building Qualifications
Suggestions and recommendations for hourly models
Revisiting other requirements in terms of hourly models
Tests for hourly models

Next CalTRACK working group meeting will be April 12, 2018.

1 Comment

Building Qualification Method Discussions Continue

3/26/2018

0 Comments

Week Seven CalTRACK Update

A quick update for this week and a reminder of the working group meeting:
Thursday, March 29th at 12:00 (PST) in which we will cover:

Final discussion of building qualification methods
Introductory discussion on Hourly Methods

During week seven of CalTRACK, consideration of building qualification methods continued. This week’s working group meeting will conclude the discussion of building qualification methods and launch the testing period. Comments or test results should be added on GitHub issues early next week to ensure they can be considered before proposals are finalized.

Homework:

Contribute final tests and comments for Building Qualifications on GitHub
Analyze and respond to test results and comments on GitHub
Attend the bi-weekly meeting on Thursday March 29th at 12:00 (PST)

0 Comments

Considering Metrics for Building Qualifications

3/19/2018

0 Comments

Week Six CalTRACK Update

Week six was primarily focused on the building qualification discussions and will continue to be the focus of testing and experimentation this week; this was coming off of an exciting working group meeting on March 15, 2018 linked below.

Recording: March 15, 2018 Working Group Meeting

Review of properties of intercept-only models in PRISM:
As we analyze building qualifications, it is useful to review the properties of PRISM intercept-only models to ensure they are properly treated. Here are a few characteristics of intercept-only models:
Properties:

Intercept-only models imply no significant effect of HDD or CDD on energy consumption was detected. Generally, this means that weather did not have a significant effect on the site’s energy consumption
In intercept-only models, predicted energy savings are the difference between the current year’s energy consumption and the previous year’s consumption
Significant temperature-related energy savings are not expected at sites with intercept-only models

Weaknesses:

These models are susceptible to poor savings estimates if the previous year was atypical. For example, if a resident did not live in their house for a majority of the previous year, then it may not be a good predictor of energy consumption in the current year
Intercept-only models impose an average energy consumption over the entire year. This yearly average may be inappropriate when estimating more granular fluctuations, such as daily or hourly energy consumption

Description of Each Proposed Metric:
During the upcoming week, we will use empirical testing to establish the preferred metric and threshold to determine a building’s suitability for CalTrack methods. The two proposed metrics are described below:

Coefficient of Variation Root-Mean-Square-Error (CVRMSE)
The CVRMSE is calculated by:

Measuring the distance between each predicted value and actual value
Squaring each of these distances
Averaging all of these squared distances from (2)
Taking the square root of the average

Because the distances are squared in the CVRSME before they are averaged, outliers can have a large effect on this metric. In the context of pay-for performance, we are uncertain if it is advantageous to choose a metric that is sensitive to outliers or not. We look forward to seeing test results on this issue.
Mean Absolute Percent Error (MAPE)
The MAPE is calculated by:

Subtracting each observation’s actual value from their predicted value
Dividing by that observation’s actual value
Taking the average of the value in (2) for all observations
Multiplying by 100 to give the result in a percentage

The MAPE is another appealing metric. It is worth noting that with a MAPE calculation, it is problematic if the actual values are zero for observations because this would require dividing by zero, which is a mathematical problem.

Other Reference Materials on Baseline Models that inform the discussion:
In the Granderson, et. al. study cited below, one key question it tackled was: “How can buildings be pre-screened to identify those that are highly model predictable and those that are not, in order to identify estimates of building energy savings that have small errors/uncertainty?”

Granderson, J., Price, P., Jump, D., Addy, N., Sohn, M. 2015. Automated Measurement and Verification: Performance of Public Domain Whole-Building Electric Baseline Models. Applied Energy 144:106-133.

In the Southern California Edison study, a buildings were sorted into four categories to identify applicability of the analytical method.

Southern California Edison with FirstFuel. February 2016. Energy Efficiency Impact Study for the Preferred Resources Pilot

Suggestions on Testing These Metrics
Remember, our goal for testing is to establish our preferred metric and threshold for building qualification. When testing the CVRMSE and MAPE metrics, we have some suggestions to yield the most informative results:

Test residential, commercial, and industrial buildings separately. This provides information on CalTrack’s performance across different building types
Test intercept-only model performance. This will inform model usage decisions in the future

Non-Routine Adjustments:
Some discussion arose regarding the possibility of making non-routine adjustments for sites that are outliers. CalTRACK 1.0 addressed this issue by stipulating specific criteria for accepting a non-routine adjustment. Specifically, if savings exceeded 50% +/-, either party would be able to make an appeal to remove the project from the portfolio. Other specific considerations may be related to program eligibility, such as a house that adds solar panels during a performance period. At a general level, CalTRACK methods shy away from stipulating methods for non-routine adjustments, as these tend to demand substantial additional effort and may require additional data that would run contrary to the premise of using CalTRACK methods in the first place. As CalTRACK 1.0 testing demonstrated, for aggregators of residential projects, larger sample sizes diminish the effect of these outliers.

Participant Homework:

Review the Issues page and plans for testing on building qualifications
Conduct your own tests on relevant questions
Analyze test results as they emerge and comment

0 Comments

Metrics & Model Selection for Building Qualification

3/12/2018

1 Comment

Week Five CalTRACK Update

We look forward to another working group meeting Thursday March 15th. Here are some updates from the activity last week.

Building Qualification Summary:
During the past week, we started discussions to establish an empirical method to determine a building’s suitability for CalTRACK methods. Listed below are two topics related to building qualifications that require testing:

Preferred Metric: The two main metrics being discussed to evaluate a building’s suitability for CalTRACK methods are Coefficient of Variance Root-Mean Standard Error (CVRMSE) and Mean Absolute Percent Error (MAPE). The ideal metric will effectively determine a building’s suitability for CalTRACK methods, while also being convenient for other stakeholders in pay-for-performance programs.
Model Selection: Due to the tendency of buildings that are unqualified for CalTRACK methods to be intercept-only models, it is proposed that the quality of intercept-only models is evaluated based on the variance of mean usage for daily or billing period data instead of the R-Squared criteria.

Some additional areas that require further analysis before our discussion of building qualifications closes:

How do our metrics and methods align with pay-for-performance programs? Do they reflect performance risk and uncertainty?
What should we do with disqualified buildings? What is the best way to accommodate them in pay-for-performance programs?

The discussion of building qualifications is scheduled to close on March 14th, so remember to add proposals, comments, and test results as soon as possible.
Homework for Participants:

Attend the standing, bi-weekly meeting on Thursday, March 15th at 12:00 PST
Review Github and provide feedback on any relevant issues. This will be the last week for suggesting building qualification test plans.

1 Comment

<<Previous

Forward>>

Archives