[BUG FIX] Update your environment to CityLearn==2.1b12

NeurIPS 2023 Citylearn Challenge
, ,
#1

TL;DR

  • To apply the bug fixes, update your local evaluation requirements.txt file to use CityLearn==2.1b12 and also pip install CityLearn==2.1b12 in your local evaluation environment.
  • The online evaluator has been updated to use CityLearn==2.1b12 .
  • All submissions to Phase 2 will be rerun with CityLearn==2.1b12 to update the leaderboard by 2023-10-23.
  • The bug fixes will affect results for control track but not forecast track. However, both tracks participants are advised to make the update.

Summary of bug fixes

We recently made some bug fixes that will affect the KPI values of any Phase 2 local or online evaluation using CityLearn<=2.1b11 in the control track only. These bug fixes are:

  1. A bug introduced when transitioning from CityLearn==2.0.0 to CityLearn==2.1b1 affected how the dhw_storage and electrical_storage were charged. The round_trip_efficiency was being multiplied by both energy_init and energy instead of the latter alone. This caused the charged/discharged energy to be smaller than required which will affect net_electricity_consumption in the buildings. This was the original change that caused the bug and this was the change in CityLearn==2.1b10 that fixed the bug. Thanks to @rl11 for spotting this bug.

  2. A bug introduced when transitioning from CityLearn==2.1b9 to CityLearn==2.1b10 affected how POWER_OUTAGE_NORMALIZED_UNSERVED_ENERGY_TOTAL was calculated. In an attempt to avoid a zero division error when calculating this KPI for simulations without a power outage, a bug was introduced that considered unserved energy during both outage and non-outage time steps instead of the former alone. This caused very low KPI values thus, affecting the average score of all KPIs. This was the original change in CityLearn==2.1b10 that caused the bug and this was the change in CityLearn==2.1b12 that fixed the bug. Thanks to @callum for spotting both the zero division and low value bugs.

  3. An old bug introduced when transitioning from CityLearn==1.1.0 to CityLearn==1.2.0 where cooling_storage, heating_storage and dhw_storage were causing slightly overestimated electricity consumption when the energy to be added to any of these storage systems as a result of a charge action exceeded the capacity. Internally, the storage system models accounted for the excess energy however, the cooling_device, heating_device and dhw_device used to charge these storage systems used the energy before adjustments for capacity limits to calculate their own electricity consumption. Note that this bug did not affect electrical_storage. This was the pull request that fixed the bug. Thanks to @skywuuuu for leading me to back calculate net_electricity_consumption that led to finding this old issue.

What you should do

  • To apply the bug fixes, update your local evaluation requirements.txt file to use CityLearn==2.1b12 and also pip install CityLearn==2.1b12 in your local evaluation environment. The online evaluator has been updated to use this new version.
  • All submissions to Phase 2 will be rerun with CityLearn==2.1b12 to update the leaderboard by 2023-10-23.
1 Like
Clarifications needed, also potential bug?