Currently, I see that round 2 has only 6 procgen games instead of 16. Is that temporary or will that be until the end of round 2?
Hello @quang_tran
We plan to run the submissions on 6 public and 4 private environments. Towards the end of the round, we will run the top submissions on all 16 public environments and the 4 private environments. We will soon share more details on this.
1 Like
Hello @jyotish
If you want to save computational resources, you should reduce the daily limit on submissions, rather than changing the competition schedule.
By the way, how can we evaluate the submissions in the generalization track?
Totally agree with @Feiyang, as a note for future competitions - would be much better to limit submissions to 1/day, and allow more resources per submission.
Sample efficiency competition would be more interesting with more GPU time allowed.
Agree, came across many sample efficient algorithms that require more than 2 hours to run 8M steps
Thank you for the information
I am humbly against Feiyang. Not everyone gets that amount of computation resources to try out their ideas. A decent amount of daily limits on submissions are helpful.
On the other hand, I agree with @jurgisp and @quang_tran that we should relax the 2-hour limit. I think this makes the competition bias to on-policy algorithms. On-policy algorithms can take advantage of large batches and therefore use less training iterations. But off-policy algorithms usually work with a much smaller batch size and require more training iterations and more time to train.
1 Like