Change in daily submission limits for round 1

Hello,

We are decreasing the daily submission quota to 2 submissions per participant due to reduced cluster capacity.

EDIT: The quota is now restored to 5 submissions per day.

1 Like

somewhat related question:
Are top50 from warm-up round really getting 10k$ in AWS credits as advertised in the email all participants received?
This would be awesome!

2 Likes

@jyotish or @mohanty with the submission limit being dropped to 2 per day, is it possible to relax the evaluation time limit a bit? Maybe from 30 min to 45 minutes or an hour? I find that miner and big fish are susceptible to longer episodes and the evaluation time limit can be hit even with a model that performs well on the other environments.

Edit: I actually did some measurements and have some data that might be interesting. Bigfish is especially problematic. Myself as a human player was finding that I could beat it in approx 600-700 timesteps. I timed the rollout.py script and found that a baseline impala ppo model was giving me a throughput of ~450 timesteps per second. As far as I know, the rollouts are performed sequentially and not using multiple workers. This means that an optimal human level performance would equate to:

(1000 episodes * 650 steps) / (450 timesteps per second) = ~24 minutes.

This is under the 30 minute time limit but it’s pretty close. A better model could actually take longer and fail due to the 30 minute limit.

I think it makes sense and would be fair for the evaluation limit to at least be long enough to handle a baseline model running the eval episodes num (1000) * the max steps per episode (1000) / baseline model throughput (450) which is ~37 minutes. It’s a small increase, but it could make a difference for some of the failures we’re seeing. 45 min could be nice for algos that might need a bit more computation than the baseline.

4 Likes

Do you know when the quota would be restored as before to 5 submissions per person per day?

1 Like

I did my last submission about 8 hours ago, but the site is saying that I can only submit again at 8 aug 8am UTC, 36 hours from now, is this correct?

Hi @ielmorla,

Can you please share where did you see 8th Aug 8 AM message?

The limit is running window, so your next submission time should be 24 hours plus to your second last submission. (verified the configuration too just to be sure)

Hi @victor_le,

We are working on restoring the limit on top priority, but you can expect higher limit somewhere in ~1-2 weeks in worst case. We will keep you all informed as soon as possible on situation.

2 Likes

Hi @tim_whitaker,

Thanks for the suggestion.

We agree with your observation and have shared across the concern with the organizing team, and awaiting a decision on it. Although based on initial discussion, the limit will be increased – just collecting a few more datapoints from incoming submissions.

2 Likes

Thanks @shivam. Curious if there’s any update on the evaluation time as of this week?

Hi @tim_whitaker,

You can expect an announcement by ~Tuesday about it along with new surprises! :smiley:

3 Likes

Hello all,

The daily submission quota is back to 5 per day :smile:

4 Likes

Hello @tim_whitaker

The timeout for rollouts is now 45 minutes (+ buffer).

2 Likes