Hello,
We are decreasing the daily submission quota to 2 submissions per participant due to reduced cluster capacity.
EDIT: The quota is now restored to 5 submissions per day.
1 Like
somewhat related question:
Are top50 from warm-up round really getting 10k$ in AWS credits as advertised in the email all participants received?
This would be awesome!
2 Likes
@jyotish or @mohanty with the submission limit being dropped to 2 per day, is it possible to relax the evaluation time limit a bit? Maybe from 30 min to 45 minutes or an hour? I find that miner and big fish are susceptible to longer episodes and the evaluation time limit can be hit even with a model that performs well on the other environments.
Edit: I actually did some measurements and have some data that might be interesting. Bigfish is especially problematic. Myself as a human player was finding that I could beat it in approx 600-700 timesteps. I timed the rollout.py script and found that a baseline impala ppo model was giving me a throughput of ~450 timesteps per second. As far as I know, the rollouts are performed sequentially and not using multiple workers. This means that an optimal human level performance would equate to:
(1000 episodes * 650 steps) / (450 timesteps per second) = ~24 minutes.
This is under the 30 minute time limit but it’s pretty close. A better model could actually take longer and fail due to the 30 minute limit.
I think it makes sense and would be fair for the evaluation limit to at least be long enough to handle a baseline model running the eval episodes num (1000) * the max steps per episode (1000) / baseline model throughput (450) which is ~37 minutes. It’s a small increase, but it could make a difference for some of the failures we’re seeing. 45 min could be nice for algos that might need a bit more computation than the baseline.
4 Likes
Do you know when the quota would be restored as before to 5 submissions per person per day?
1 Like
I did my last submission about 8 hours ago, but the site is saying that I can only submit again at 8 aug 8am UTC, 36 hours from now, is this correct?
Hi @ielmorla,
Can you please share where did you see 8th Aug 8 AM message?
The limit is running window, so your next submission time should be 24 hours plus to your second last submission. (verified the configuration too just to be sure)
Hi @victor_le,
We are working on restoring the limit on top priority, but you can expect higher limit somewhere in ~1-2 weeks in worst case. We will keep you all informed as soon as possible on situation.
2 Likes
Hi @tim_whitaker,
Thanks for the suggestion.
We agree with your observation and have shared across the concern with the organizing team, and awaiting a decision on it. Although based on initial discussion, the limit will be increased – just collecting a few more datapoints from incoming submissions.
2 Likes
Hi @tim_whitaker,
You can expect an announcement by ~Tuesday about it along with new surprises! 
3 Likes
Hello all,
The daily submission quota is back to 5 per day 
4 Likes