Dear Organizing Committee of NeurIPS 2022 CityLearn Challenge:
On behalf of our team, I’d like to propose the following concern about the fairness of the current competition.
According to your announcement issued from “Removal of logs from Phase 2 runs” on Sep 14, we are quite worried that some teams may have the private data logged from Phase 2 and take them for training or other usage. If someone uses these leaked data and over fits the logged data, the score will be significantly improved.
Therefore, we’d like to present the following suggestions to the organizers, which may help to improve the fairness of the competition.
(1) It would be helpful to announce again that all the teams can not use the private data logged in phase2 for training or inference in the final selected submissions.
(2) The winner teams need to release the final code (training, inference codes, etc.) to ensure the reproducibility of the final best result, which should not rely on the private data logged from Phase 2.
Thanks for your attention and wish the competition a great success.
The competition official rules have clearly stated that the participants should release their code under an open-source license. The code should be reproducible (especially for the training process). This can prevent some teams from using the illegal logged data in schema 2 of (6~10)/17 buildings.
So there is no need to worry about this problem.
Please refer to: AIcrowd | NeurIPS 2022: CityLearn Challenge | Challenge_rules
The details are as follows.
6. Prizes
To be eligible for the prizes, participants will have to release the code to their solutions under an open-source license of their choice. The submitted code is expected to be reproducible and should produce the same score on the leaderboard.