Hey @slopez!
- Why are environments grouped into tests?
All the environments in the same test have the same parameters: height, width, number of agents… the only difference is the malfunction rate. The performance of your submission is evaluated one Test at a time, and you need to have on average 25% of the trains reaching their destination to move on to the next Test. We’re interested in seeing up to what size a submission can perform well enough.
- From here, I infer that the only difference between levels in a same test is the malfunction interval, and tests differ by all other parameters, is this correct?
Correct! well not all the other parameters change, eg max_rails_in_city , malfunction_duration etc are all constant across all the levels.
- While evaluating locally, it seems that the evaluator service picks a random environment from all the environment tests. How does that work in the online evaluation, if the list of tests is infinite?
From 🚂 Here comes Round 2! (deadline: Nov 6) :
Note: Now that the environments are evaluated in order (from small to large), you should test your submissions locally in the same conditions. You can use the
--shuffleflag when calling the evaluator to get a consistent behavior:
flatland-evaluator --shuffle False
- Is the score computed on all evaluated environments, independent from which test they come from?
The final score is the sum of the normalized return across all the evaluated environments, yes. So, 0.5 point in an environment in Test_0 is worth as much as 0.5 point in an environment in Test_30.