Announcements on submissions for Round 2 and final results

NeurIPS 2021 - The NetHack Challenge
#1

Hi Everyone,

Apologies for all the confusion about round 2, this post is meant to clarify all your doubts and make some announcements.

  1. How many submissions are allowed in Round 2?
    All teams are allowed 3 successful submissions for Round 2, in case of errors in the code up-to 10 failed submissions are allowed (not valid if a score is generated but is a poor score), these will be used for the final evaluations, if 3 submissions are not received for a team, we’ll use the selected submissions from Dev Phase. Effectively only 3 submissions will be used for the final scoring, (not 6 unlike some previous posts I have posted).

  2. Why are submissions still running on 512 rollouts
    The final evaluations will be re-run on 4096 rollouts, no code changes are needed from the participant’s side. This is because the original intention was to only allow selected submissions from Round 1 and not take new live submissions during Round 2. The submissions during Round 2 provide extra time to further develop your solutions.

  3. The leaderboard is not visible
    The final results will be announced at NeurIPS, participants can still see the scores of submissions made by their team and will also have access to the scores from 4096 rollouts once they are re-run.

  4. Is it allowed to change the selections from the Development Phase now?
    The best approach would be to submit 3 new submissions during round 2. These can run same code as any older submission. If for any reason a team is unable to make submissions now and also wants to select older submissions from dev phase, please contact the organizers.

1 Like