Hello!
We have noticed that some submissions alter the control flow based on the environment name provided. This environment-specific logic is against the spirit of the competition. We expect the solutions to have a generic approach that works in the same way for all the environments. Any solution with environment-specific logic will be invalidated on manual code inspection.
Edit: Using the environment name to determine min/max rewards is fine.
hello@jyotish, as you said,
Will these submissions’ socres be re-evaluated? And reranking the learderboard?
Some teams in the warmup run used the forbidden marks, and got a ‘wonderful’ ranking in the leaderboard. It doesn’t matter.
However, in the first and second run, the scores and the ranking really maters. Thus, I think the submition using environment-specific logic with ‘wondeful’ ranking should re-evaluated, and the ‘wondeful’ ranking should be invalid.
Hello @lars12llt
Thanks for expressing your concerns. The submissions using environment specific logic won’t be considered at all for the leaderboard.
One more thing I’d like to point out is the file size limit set for scraping, 30 MB is actually quite high given Impala baseline is only ~7 MB even with the optimizer state, without optimizer its ~3 MB I guess.
ok, I was thinking of implementing some kind of environment-specific logic, I’m glad I read this before wasting my time!