Clarifications needed, also potential bug?

Control Track: CityLearn Challenge
, ,
#1

Hi, after reading through the control track in detail, we’re seeking several clarifications and also potentially found a bug in the code.

  1. What are the weights for the different reward metrics? The code disagrees with the AIcrowd description. E.g. the grid control weights (w3) are 0.075 each in the code but 0.125 on the website.
  2. The website has mentioned that the final evaluation will use different seeds for the power outages than Phase II online evaluation (is this correct?). However, will all the other environmental conditions (e.g. weather, buildings) be the same as Phase II online evaluation? We did not find a definitive answer on the website. Additionally, is the LSTM dynamics model for buildings 1/2/3 held constant across Phase II online/offline evaluation as well as final evaluation? In general, it would be helpful if all differences between Phase II offline, Phase II online, and final evaluation could be summarized.
  3. It is mentioned that Phase II online evaluation uses different weather and more buildings than Phase II offline evaluation, which only requires cooling but not heating. We’d just like to confirm that Phase II online evaluation and the final evaluation will also not require heating right?
  4. We’re fairly sure that there’s a bug in the battery charging code. Around lines 570-571, the self.round_trip_efficiency multiplier is erroneously applied in the more recent commit to both self.energy_init as well as energy, when it should only be applied to energy alone (as in the previous version of the code). This could have implication for model-based methods?

Thanks for putting together the competition!

#2

@rl11 thanks for raising all these important points. Here are my responses:

  1. The competition page is outdated and needs to be updated with the correct information especially information about the weights control KPIs. The correct weights are what is in the code. We will work on updating the competition page (@snehananavati please, can you update the pages)?

  2. Weather will be the same in final evaluation but 3 other buildings are added. This ‘final’ evaluation is already taking place. It is the ‘CityLearn Private’ you see in your submission which you cannot see the scores for. LSTMs don’t change at any point. Most of these differences are highlighted in this post.

  3. There is no heating at any point in the competition.

  4. This is a very good catch, thank you! I will check it out ASAP.

#3

Bump. What’s the status on #4? I see it fixed in the latest version of citylearn library, but will version 2.1b11 be used for this competition?

#4

It has been resolved. See this post. The competition pages have also been updated with current information about weights and KPIs.