Why this user can submit 42 times?

NeurIPS 2020: MineRL Competition
#1

https://www.aicrowd.com/participants/minerl_dqfd_baseline This user has submitted 42 entries. But the rule says we can submit twice a week.

#2

Hi @youkaichao,

Sorry for any confusion caused.
But this account belongs to MineRL organising/baseline team, not participant, so nothing to worry about. :smile:

#4

@shivam

Ok, got it.

I find it very hard to participate in this competition, the submission template just keeps raising exceptions and there are annoying bugs like https://github.com/minerllabs/minerl/issues/368 .

Since you are tuning baselines, do you mean that we should not start until the baselines are ready?

#5

@youkaichao

I apologize for down-talking the issue you have risen (which is valid), but the environment works for training and testing purposes for us (but we do not use the seed function). Similarly the submission template works as-is, which I already submitted twice as a test successfully.

I recommend you join the Discord channel linked on main competition page. There are people there that can help you with questions.

#6

@Miffyli “but we do not use the seed function”
That’s strange. The current submission template uses seeds here and it fails as-is. Which submission template do you use?

1 Like
#7

Ah alright, that’s a good observation that you could raise in the submission template as well. I use my own scripts to run lighter evaluations (not full ones) to only see if the Python code runs correctly, and then assume rest of the full evaluation works as intended on the server.

#8

Do you mean it is not necessary to use the submission template?

#9

I suddenly realized that there are 4 baselines and @Miffyli and @shivam 's submissions in the leaderboard. Do I miss something? Why I see no one (other than the organization team) has made a successful submission?

I think it would be great if failed submissions do not count so that we can feel relieved to make a trial.

#10

@youkaichao

You need to use the submission template, but evaluate_locally.sh is only used for testing if the code runs correctly on your local machine (judging by the script name). I use something similar which just skips most of the steps in evaluate_locally.sh, boots the image and runs run.py.

As for why no other submissions: There is still time in Round 1 and people usually start slow with the submission, based on observations on previous competitions. I am willing to bet we will see bunch of participants soon enough :slight_smile:

#11

So I think we should start after the baselines are tuned, right? Because I expect tuned baselines with non-zero rewards will be released so that people can work to improve upon the baselines.

#12

I highly recommend you familiarize yourself with the environment as soon as possible, perhaps try your own ideas on the simpler tasks (TreeChop). This way you should not run into bigger roadblocks when you play around with the baselines, taken from the Discord channel.

There are some baselines available from other players already: