It would have been nice if only graded submissions were considered in the daily limit.
Thank you for the response, and it makes sense.
However, I can see scenarios where you distinguish between fails like the attached screenshot here, and other runs which failed after training initiated successfully and the user received some feedback:
Simply put, I don’t see the users getting any feedback/information back unless the training is complete, thus, it seems safe not to include failed builds and failed trainings in the total number of submissions.
@amirabdi: Agree ! But the decision is a site wide policy decision to prevent abuse of the setup by participants (either to game the system to squeeze our more probes, or also in some cases to overload the evaluation system just before the deadline). At the moment, the site wide policy decision will unfortunately stay the same.
But, if you make a case with the organizers, the number of daily submissions allowed could very well be increased if the organizers (who eventually are the compute providers) are onboard !
We are also working on releasing a few features which will allow you to better debug your submission before making the actual submission. Please expect a few updates on the same soon.
Cheers,
Mohanty
1 Like