How many times of the episode will the submitted agent been tested for?

NeurIPS 2022: CityLearn Challenge
#1

While local evaluating the customized agent, I noticed in the default settings, it’s run for 3 episodes.
How many episodes would the submitted model been evaluated?

1 Like
#2

Thanks for your question @blanck_smith and I am happy to clarify. In your local evaluation, you can run for as many episodes as needed to train your agent. The online evaluation runs 1 episode, twice in parallel and takes the average score of the parallel runs (@dipam, please confirm).

#3

Thanks for your answer!

#4

Hi @blanck_smith

Making a small correction to what @kingsley_nweye mentioned. The online evaluator runs 5 episodes in 2 parallel runs. So a total of 10 episodes are run and averaged.

1 Like
#5

@kingsley_nweye Can you verify that this is the intended procedure (run 5 episodes). The problem is that each episode is exactly the same and that makes it beneficial to overfit the first episode.

#6

Hi @mt1 thanks for your question. Please, @dipam can you provide an answer to this question about online evaluator runtime?

1 Like