Can you fix the ambiguity? You are using CROWDAI_IS_GRADING (https://github.com/minerllabs/competition_submission_starter_template/blob/0bac0b99d217a5765a6ff90470e5745a24fe92af/utility/docker_run.sh#L36) and AICROWD_IS_GRADING (https://github.com/minerllabs/minerl/pull/169/files)
@kashin: That is an artifact left from the transition from crowdAI to AIcrowd. And on all evaluations on AIcrowd, we set both the variables CROWDAI_IS_GRADING and AICROWD_IS_GRADING to True.