❗️Important Updates for Phase II of CityLearn Challenge 2023

snehananavati · September 19, 2023, 9:22am

TL;DR

Fixed negative heat pump action bug in RBC.
CityLearn updated to support power outage scenarios (see comparison to Phase I environment).
Control track evaluation updated to consider performance during power outage.
Online evaluator dataset to change in Phase II.
Three buildings used in Phase II public leaderboard evaluation.
Private leaderboard at the end of Phase II that uses six buildings will be used to select winners.
From September 19, 2023, when Phase II commences, pull the latest control and forecast track starter-kits for your local evaluation and rerun pip install -r requirements.txt.
There may be breaking changes in CityLearn and the starter-kit that affect your solutions from Phase I.
Ensure that your submission to the the private leaderboard at the end of Phase II can be used on a larger number of buildings than trained on.

What has changed in Phase II?

Please take note of the following changes as we advance to Phase II of The CityLearn Challenge 2023:

Bug squashed!

There is a bug in CityLearn==2.0b9 used in Phase I where the RBC agents provide negative actions to the cooling_device. This causes the building’s cooling_demand and cooling_electricity_consumption to be negative. Since a large proportion of building loads are attributed to cooling, the building’s net_electricity_consumption becomes negative as well as KPIs that make use of it in their calculation. This bug has been fixed since CityLearn==2.1b1.

Lights out!

See comparison between Phase I and II environments.

CityLearn now supports power outage scenarios! During a power outage event, the grid is unable to provide the buildings with electricity and control solutions can only make use of the available distributed energy resources in buildings including domestic hot water storage, battery and PV system to satisfy cooling, domestic hot water and non-shiftable loads otherwise, risk thermal discomfort and unserved energy during the event. During normal operation i.e., when there is no power outage, there is unlimited supply from the grid.

The outage signals are either defined in the building data file or generated using some stochastic model that is defined in the schema for each building and constructed at runtime.

The CityLearn Challenge 2023 makes use of the stochastic model approach that is based on Reliability Metrics of U.S. Distribution System. The stochastic model generates time series of power outage signals using System Average Interruption Frequency Index (SAIFI) and Customer Average Interruption Duration Index (CAIDI) to find outage days, their start time and duration. The generated signal for a fixed SAIFI and CAIDI is controlled by changing the model’s random_seed. If interested, please, see the docs for how the model is implemented and feel free to suggest any improvements .

By the way, power outages will only be applied to the control track environment so, you do not need to worry about blackouts in your forecast track submissions and evaluations . Participants in the control track should expect outage events to happen at the same time in all buildings, at least one power outage event, and events that last at least, half a day during online evaluation.

How well did you control the dark?

A new score, Score_Control^Resilience has been added to the control track to evaluate the average building resilience during power outage events. This score is the average of two KPIs: 1 - thermal resilience (M), and normalized unserved energy (S). See the competition page for their definitions.

With the addition of Score_Control^Resilience, there are four control scores: thermal comfort score (Score_Control^Comfort), an emissions score (Score_Control^Emissions), a grid score (Score_Control^Grid), as well as the resilience score (Score_Control^Resilience) and their respective weights are defined here.

Who’s topping the leaderboard?

In Phase I, participants in both the forecast and control tracks were given the same dataset consisting of three buildings and one month worth of data. The online evaluator made use this same dataset to evaluate submissions and update the leaderboard. Thus, the score calculated during local evaluation were the same as those posted on the leaderboard (assuming the same CityLearn environment version in both local and online evaluation).

In Phase II, the same three-building dataset is provided for local evaluation in both tracks but with an updated schema.json that defines the stochastic power outage model for in each building as well as a random_seed that is used to generate outage signals. Participants can change this random_seed using the provided helper function in the control track starter kit (local_evaluation.update_power_outage_random_seed) to train their control agent(s). Although the forecast track uses this updated schema, power outages will be disabled in its environment.

On the online evaluator side in Phase II and prior to the end of the competition, a completely different dataset is used where there are three buildings (including the three in the local evaluation dataset) and three months worth of data based on a different weather file. Thus, submitted agents should be able to generalize to weather conditions, occupant behaviors, and buildings not seen during training. Also, in the control track, a public leaderboard will be reflective of the average scores from using three different private random_seed values to generate stochastic power outage signals in the environment. These three seeds will be kept constant throughout Phase II. In the case of the forecast track, the public leaderboard is reflective of just one environment’s score since power outages are not considered hence no stochastic influence on the environment by random_seed.

At the end of Phase II, submission acceptance will cease and a private leaderboard that is only visible to the challenge organizers will be launched. This leaderboard will be used to select winners for both tracks. This private leaderboard is reflective of a similar dataset as that used for online evaluation during Phase II except:

It is made up six buildings, three of which are used in Phase II public leaderboard evaluation.
The control track private leaderboard will be reflective of the average scores from using three private random_seed values that are different from those used in Phase II public leaderboard.

Hence, participants must make sure their final submission at the end of Phase II is coded in such a way that it is applicable to more buildings than it was trained on.

What you should do …

From September 19, 2023, when Phase II commences, pull the latest control and forecast track starter-kits for your local evaluation and rerun pip install -r requirements.txt.
There may be breaking changes in CityLearn and the starter-kit that affect your solutions from Phase I.
Ensure that your submission to the the private leaderboard at the end of Phase II can be used on a larger number of buildings than trained on.

snehananavati · September 20, 2023, 5:09am