Has the submission process changed for round 2?

Unity Obstacle Tower Challenge
#1

I tried re-submitting my agent from round #1 to see how it did on round #2 - but get the following error

did something change? (all i did was push a text change and re-tag my repro)

   The following containers terminated prematurely. : agent
    Please contact administrators, or refer to the execution logs.

https://gitlab.aicrowd.com/joe_booth/obstacle-tower-challenge/issues/119

#2

Hi @joe_booth

Looking at the logs, it seems that you will need to upgrade the ObstacleTowerEnv python api to version 2.0 for the submission.

1 Like
#3

@arthurj

Did the evaluation of the submissions change for round 2?

Like what reset parameters are used and how many random seeds?

#4

Hi @Leckofunny

The evaluation protocol is the same as from Round 1. So submissions are tested on five held-out seeds. We also use the same reset parameters from Round 1, with the exception of cases where there are now additions, such as the new additional themes on higher floors, or additional floor layouts.

#5

@arthurj in other words, can we assume the ‘config=None’ for the evaluation environment?

#6

@hanschoi86

That is correct!