[ANNOUNCEMENT] Submissions Open

[mlerik]

Hi All

We are now all set for your submissions.
Please follow the instructions in the starter-kit to get your submission started.

ATTENTION for Windows users: Please use WSL to avoid windows dependencies. We will update you as soon as we have another solution ready.

The Flatland Team

[mugurelionut]

I made a debug submission and it failed with an error at this line from run.py:

observation = remote_client.env_create(obs_builder_object=…)

The actual error is missing to find a file locally:
2019-07-31T23:59:33.880474871Z FileNotFoundError: [Errno 2] No such file or directory: ‘Test_0/Level_0.pkl’

How does that work? Why would a “remote_client” try to look for a test file locally?

In hindsight, these instructions regarding how to run things locally look fishy:
export AICROWD_TESTS_FOLDER=<path_to_your_tests_directory>
python run.py

Why should run,py (which is part of my repository / Docker image) need to have access to the folder with tests that the evaluator uses?

That said, following these instructions I was able to run things locally correctly. But when making the submission, I guess nobody sets this AICROWD_TESTS_FOLDER variable. Who is supposed to do it? And, in general, as I said, this seems wrong. Even if I do set it to something, how will that folder get the tests used for evaluation?

What am I missing? / What do I need to do to not get that FileNotFoundError anymore?

[mohanty]

Hi @mugurelionut,

All good questions.
I just went through your submission, and it looks like you are using an older version of the starter kit ?
https://gitlab.aicrowd.com/mugurelionut/flatland-challenge-starter-kit ?

And coming to your question of why does the remote_client need access to the test files locally, that was indeed a non intuitive design decision that we had to take, because we want to lets participants have access to a whole RailEnv object to do whatever they want with it, but we did not want participants to arbitrarily change the state of the environment (for instance, a participant could simply loop over all the agents, and set their position to the target position, and the env would be done in a single step ), which would affect our ability to reliably measure the score for all the participants.

So what we instead do now, is run two copies of the same environment, one inside the container running the user-submitted code, and one inside a separate trusted container. When you do an env_step on the remote_client, we apply that step on both the local instance of the environment, and the “remote” instance of the environment in the trusted container.

We compute things like the total rewards, when an env is done, etc from the env instance in the trusted container (remote), and we compute observation from the env instance in the container running user code (which also allows us to safely let users design their own observations).

Now, to make this happen, we can ofcourse expose the RailEnv (and all its interfaces) in the remote container as an RPC, adding enough safe guardings to ensure it cant be abused. But that will cause a lot of data flowing back and forth between the containers over the network on all interactions. For instance, the env state objects for larger grids (100x100 or more) can be in hundreds of megabytes, and that would hugely affect the performance during the evaluation.

He we just securely mount a copy of all the test envs in both the containers, and when you do a env_create call, the remote service sends you the “key” of the test-env that its loading at its end, and the remote_client also locally instantiates a copy of the same env, which you of course can access by : remote_client.env (ref) .

In your case, the only reason you could be gettting this error, is either if you have not downloaded the tests, or placed them incorrectly, or passing a wrong value to the AICROWD_TESTS_FOLDER env var.

The AICROWD_TESTS_FOLDER env variable should point to the root of the directory which has the individual test sets. If say, my directory containing the tests is called as test-env, then its structure should look like :

└── test-envs
    ├── Test_0
    │   ├── Level_0.pkl
    │   └── Level_1.pkl
    ├── Test_1
    │   ├── Level_0.pkl
    │   └── Level_1.pkl
    ├── Test_2
    │   ├── Level_0.pkl
    │   └── Level_1.pkl
    ├── Test_3
    │   ├── Level_0.pkl
    │   └── Level_1.pkl
    ├── Test_4
    │   ├── Level_0.pkl
    │   └── Level_1.pkl
    ├── Test_5
    │   ├── Level_0.pkl
    │   └── Level_1.pkl
    ├── Test_6
    │   ├── Level_0.pkl
    │   └── Level_1.pkl
    ├── Test_7
    │   ├── Level_0.pkl
    │   └── Level_1.pkl
    ├── Test_8
    │   ├── Level_0.pkl
    │   └── Level_1.pkl
    └── Test_9
        ├── Level_0.pkl
        └── Level_1.pkl

And in your case, lets say, you set the AICROWD_TESTS_FOLDER as /your/path/to/tests/folder,
then can you ensure that you can get this following snippet of code to run :

import os
TESTS_FOLDER = os.getenv("AICROWD_TESTS_FOLDER", "/tmp")
test_path = os.path.join(TESTS_FOLDER, "Test_0/Level_0.pkl")
if not os.path.exists(test_path):
  raise Exception("Could not access test file at the right location :( ")
else:
  raise "File structure all correct !!  Env variables set correctly !! Finally :D !"

Do note that on the evaluator, this env variable will be automatically set, so please do not try to overrride that env variable in your run.sh file.

Cheers,
Mohanty

[mugurelionut]

Thanks. It seems I was indeed using an older version of the starter kit (in particular, I hadn’t updated environment.yml to its new version - once I did that I could get a running debug submission).

That said, the 1st debug submission which actually ran the tests, eventually failed. I guess that’s because on at least one of the tests I had a more than 15 minutes timeout (I don’t remember where I read that this was a constraint). The instructions say to tag aicrowd-bot on the issue to get some of the relevant logs which are useful for debugging. Stupid question, but what does it mean to tag aicrowd-bot on an issue? Does it mean adding an aicrowd-bot label to the issue? Or is this some git-specific terminology? (sorry, I haven’t used git before)

[mohanty]

@mugurelionut: We had pasted some logs, and investigating why your latest submissions are failing. The 15minutes timeout one is a good direction, and yes we do have that constraint, where the submissions have to send some signal every 15minutes. (You could in principle even send a no-op ).

[mugurelionut]

Hi @mohanty,

Can I also get access to logs for my latest 2 debug submissions? (#10245 and #10246) They’re really about me not knowing how to setup some things properly in the environment (you can read what I commented on them, I don’t want to get into more details here). Of course, things work fine when running tests locally on my machine.

[mugurelionut]

Never mind. I figured things out with a bit of trial and error.