Is it a preknowledge that we should select one of existing submissions for the final evaluation?

vrv · November 12, 2020, 7:57pm

I understand your frustration, especially after coming this far in the competition and feeling disappointed

However, participants were made aware that they can only make submissions for Warm-up round, Round 1, and Round 2. I am pasting below the same content from the challenge overview page.

ROUND 2 (FINALS)

Round 2 will evaluate submissions on the 16 public Procgen environments, as well as on the 4 private test environments. Participants’ final score will be a weighted average of the normalized return across these 20 environments, with the private test environments contributing the same weight as the 16 public environments.

This round will evaluate agents on both sample efficiency and generalization.

Sample efficiency will be measured as before, with agents restricted to training for 8M timesteps.

Generalization will be measured by restricting agents to 200 levels from each environment during training (as well as 8M total timesteps). In both cases, agents will be evaluated on the full distribution of levels. We will have separate winners for the categories of sample efficiency and generalization.

Because significant computation is required to train and evaluate agents in this final round, only the top 50 submissions from Round 1 will be eligible to submit solutions for Round 2. The leaderboard will report performance on a subset of all environments, specifically on 4 public environments and 1 private test environment.

The top 10 submissions will be subject to a more thorough evaluation, with their performance being averaged over 3 separate training runs. The final winners will be determined by this evaluation.

Round 2 marks the end of the competition in terms of submissions. The final evaluations that we run are only to make sure that the results are consistent across different runs.

Normally, we would pick the top 10 entries from the leaderboard and use them for the final evaluations.
Based on discussions from Round 2 is open for submissions 🚀 - #3 by karolisram, we decided to let the participant choose their best submission (which probably would be the best scoring one on the leaderboard) instead of us taking the best scoring submission.

I hope this helps.