My submission #284265 stopped at the beginning of the evaluation stage and returned the message: “Gym server stopped unexpectedly. Please contact the admins.” Could someone assist me with this issue? Thank you!
@thmei02 are you referring to a different submission by any chance? The one you mentioned says that the evaluation has timed out.
Thank you for your response! I’m a bit confused by the inconsistent log messages shown on the public and private submission pages. On the public submission page, it says “Gym server stopped unexpectedly. Please contact the admins,” while on my private submission page it says “Evaluation timed out 
. Please ensure that your code runs within the specified compute budget.” Are these two errors related, or do they indicate different issues?
Additionally, is there any guidance on what is considered an acceptable Avg Turn Time to avoid the “Evaluation timed out” error? My current average turn time is around 37 seconds with a batch size of 8, which I think is relatively low but it still fails.
Thank you very much for your help!
Ahh I see, thanks for bringing that up.
There should not be two different error messages, we will get that fixed soon. 
For the timeouts, there are two types of timeouts.
- Per batch generation timeout set at
(batch_size * 10)s - Overall evaluation timeout set at
7200s
The error you are seeing is an outcome from breaching the 7200s overall timeout.
Thank you for your reply!
However, I am quite confused by the two types of timeouts, as I haven’t found these constraints mentioned in the official rules (challenge rules). According to the rules, “Each example will have a time-out limit of 30 seconds and will be tested with the AICrowd submission system.” The same performance constraint — “Only answer texts generated within 30 seconds are considered” — is also stated on the challenge homepage.
Besides, I could not find any reference to a 7200-second overall timeout in either document. Do I miss some important information?
Thank you very much for your assistance!
We are waiting to hear from Meta on their intention for the overall timeout. But for now, we are bumping the overall timeout from 7200s (2h) to 14400s (4h).
For the per turn timeout, for now, the evaluator will be using 10*batch_size as the timeout. We will update the 10s multiplier to a different value based on internal baselines that Meta is working on. This is likely going to be something smaller than 10s.
We will update the challenge rules once we have a confirmation from Meta.
Thank you so much for the detailed update! I appreciate the clarification about the overall timeout policies. Thanks again for your support!
We will post an update here once the evaluators are ready with the 4h timeout.