I was wondering which tests are being reached by yours and by the highest ranking submissions. In the leaderboard GIFs I see up to five cities, so I’m guessing the top OR solutions are reaching at least test level 12. It would be useful to know at which level OR solutions are in order to decide some hyperparameters for RL solutions.
1 Like
As a quick lower bound: each test has 10 environments, and each environment awards between 0.0 and 1.0 points. So the top submissions went through at least ~270 environments, or 27 tests!
Then the harder question is: what is the average score per environment?
If you want to get an idea of how well OR solutions perform you can experiment with last year’s 2nd place solution: https://flatland.aicrowd.com/research/top-challenge-solutions.html#second-place. It is pure Python and super fast.
See here for an explanation of their approach: https://www.youtube.com/watch?v=rGzXsOC7qXg&feature=youtu.be&t=724