Hi, everyone! Could you tell - why there is a counter of failed submissions? I think, that if you have 10 submissions available per day than it should be 10 successful submissions. Otherwise, some problems with you repo take away your amount of available subm-s. Could organizers change that?
Hi @boris,
I believe there is a daily limit to help balance server load. That being said, perhaps it could be increased. @mohanty what are your thoughts?
@boris @arthurj : There unfortunately is no right answer for the question, without complicating the question a lot further.
A simple example would be, when because of some particular nuance in the problem, the evaluation fails towards the end of the evaluation. And the said participant, continues to keep making submissions because he/she is confused !
The other is, participants using the evaluation setup to “time” their evaluations. And exiting at the end.
or more sinister, where someone is just trying to hog up all the compute resources just before the end of the competition to retain their rank. (all you have to do is a raise “Exception” whenever you have hogged some resources for some time, and repeat)
So, after numerous internal discussions we went with the simple idea that “wrt the ratelimits, a submission is a submission…failed or successful is irrelevant”
That said, I am always happy to increase the number of valid submissions per day. Infact it was increased from 5 to 10 recently, and all game for increasing it to say 15 or 20 / day.
If this gets abused at any point, then we can manually intervene, but for now, I am also in favor of increasing the number of submissions per day to 20 !
@arthurj , @harperj @anhad: I am assuming you guys would be okay with this, hence would go ahead with the setup.
A note to self and the community : This number can be reduced back to 5 or 10 if this is abused, or we find it necessary for some other reason !
I think increasing the limit to 20 submissions a day sounds good to me now. If we run into issues we can decrease it to 15 or 10 in the future. @boris hopefully this will make it more straightforward for you to resolve the issues and make a successful submission.
1 Like
@boris I would also like to point out that if you ever find that your submissions have been counted because of errors on our end, feel free to tag any of the aicrowd-admins in any such submission and we can probably delete that submission for you so that you have more slots in your quota (on a case-by-case basis).
Submission failed : The participant has no submission slots remaining for today.
```
I have waited for almost two days without quotas.
https://gitlab.aicrowd.com/ChenKuanSun/obstacle-tower-challenge/issues/30Exactly same here.
They may have set the max submission for 20 overall, instead of some number per day.
So if we reach 20, do we have to abstain?
@ChenKuanSun @andwetry: That was definitely a mistake at our end. It should be 20 per day. Can you try now ?
1 Like
Thank you for raising the daily submission limit to 20.
Can the admin please clarify if there is a limit of total number of submissions for round 1 and round 2? Thanks!
1 Like
it work! thanks!!! !!!