Submission Times

Sound Demixing Challenge 2023
#1

https://www.aicrowd.com/challenges/sound-demixing-challenge-2023/problems/cinematic-sound-demixing-track-cdx-23/submissions

Why is quickpepper947 able to submit 6 times a day, which is over the 5 time limit? And my account can only submit once.

Post-challenge discussion
#2

xxx
xxx1395×842 122 KB

xxx
xxx1411×523 74.4 KB

#3

@mohanty In this thread the organizer said that the submission quotas are checked in the last 24 hour window, while we found this participant made 7 successful submissions within a same 24 hour slot. Just wondering what happened to the submission system, as our attempts to make more than 4 submissions in a day were always blocked.

#5

1
12592×770 162 KB

2
22610×244 55.5 KB

3
32546×250 54.4 KB

4
42626×502 109 KB

Again there are 7 successful submissions in the last 24 hours, and even assuming that the quotas are updated at every midnight, there are still 6 successful submissions in May 8th (I am in UTC +8, so it counts from 08:00:00), both exceeding the maximum submission quota of 5 shown in the submission page.

I mean we are completely fine if our models are not the best (this is the most common case for competitions right?), but the submission system let us feel like there are something wrong with our team, as we can only submit 4 entries per day throughout the past weeks and we thought that this is the submission policy for everyone. It is really weird to see different teams having different submission quotas and even exceeding the maximum quota shown in the system.

#6

Yeah, if some teams are getting extra submissions thats not fair.

#7

Add on to this, my team actually had ten submissions quota per day, at least before last week.
It changed to five during the last week.
image

1 Like
#8

I guess that was the same for everybody. I had 10 submissions for a while too, not sure when exactly it changed back to 5 though.

1 Like
#9

I am a team of four participants. Each of us can only submit once, and when submitting a second time we will show that we are currently unable to submit. We only have one chance per person, not 5 chances or 10 chances.

#10

Hi,

first, congrats to your great scores!

I didn’t get any extra submissions and, to be honest, had a hard time understanding how the submission quota actually works, myself. As I didn’t team up with individual accounts, the max submission count was likely different. I can’t say if aicrowd has some rounding issues when it comes to team submissions vs individual submission so maybe they can improve their docs on this part.

1 Like
#11

Hi there,

Many thanks for the reply here and congrats too for the good job on the contest - we know that if you have combined the 3 best systems for the 3 tracks you should be #1 there. We are definitely not attacking you for getting more submissions in some secret ways, our purpose is more about questioning the entire submission system, as now it seems like different teams were indeed having different numbers of submission quotas. We hope the organizers and the AIcrowd team can provide an explanation, at least provide some information about how the submission quotas were decided and, if they have been changed during the contest, the way they changed them and whether they were identical to all participants.

#12

Dear @JusperLee ,

Thank you for bringing your concerns to our attention. We have thoroughly investigated the issue and have found that there was a misunderstanding, for which we apologize.

Regarding the submission quota, we would like to clarify the following points. Prior to the extension of the deadline, the submission limit was initially set to 10 submissions per day. However, due to a high submission load that caused many submissions to become stuck, we made the decision to extend the deadline by one week. During this extension period, the submission limit was 5 per day.

Regardless of the size of the team, every team is granted 5 submissions per day (or 10 when the limit was 10). This allocation is not based on the number of team members. Therefore, if only one member of your team submits, they can utilize all 5 submissions for that day. If multiple people submit, it counts towards the entire team’s quota.

It is important to note that our platform does not have a mechanism in place to allow teams or participants to have a different number of submissions. The submission limit remains consistent for all teams throughout the challenge.

Lastly, there was a wrong communication regarding the submission window. Our platform supports two options: a Rolling 24-hour window or a Fixed window between UTC 00:00:00 and UTC 23:59:59. While the intention was to utilize the Rolling 24-hour window, an unintended bug caused the Fixed window to be used for the CDX challenge. We acknowledge that this was not accurately reflected in the forum post. Upon reviewing all the submissions made in the challenge, we can confirm that no team has exceeded the limit of 5 submissions in a day within the fixed window of quota reset (UTC 00:00:00 to UTC 23:59:59). Your screenshots will also confirm the same.

Once again, we apologize for any confusion and inconvenience caused. We value your feedback and will take it into consideration as we strive to enhance the clarity and accuracy of our communications. Thank you for your understanding.

2 Likes
#13

Hi @dipam,

Thanks for the reply. There are still few points I would like to receive more comments on, and it would be helpful if you could elaborate more:

Therefore, if only one member of your team submits, they can utilize all 5 submissions for that day. If multiple people submit, it counts towards the entire team’s quota.

I don’t know much for other teams, but we did try to submit more than 4 submissions a day (counting the total number of submissions per team) but failed due to the system only allowed 1 submission per team member. I think we can find some previous error messages when we tried to make the 5th submission if needed.

Upon reviewing all the submissions made in the challenge, we can confirm that no team has exceeded the limit of 5 submissions in a day within the fixed window of quota reset (UTC 00:00:00 to UTC 23:59:59). Your screenshots will also confirm the same.

We actually posted another comment in this thread above: Submission Times - #5 by JusperLee, where 6 successful submissions were in the submission page between UTC 00:00:00 to UTC 23:59:59 (we’d like to emphasize that we are in UTC + 8 and the webpage shows local time, so it should count from 08:00:00 in our local time zone), so the screenshots do not confirm the same. There was another comment saying that the submission quota changed to 5 during the last week: Submission Times - #7 by yoyololicon. I think more in-depth investigation might be important and necessary to reconfirm whether the submission system was functioning well.

#14

@XavierJ , thanks for the rebuttal post. Indeed some deeper investigation was required on my part, and now I believe I’ve found the root cause of the issue.

Along with the previously mentioned setting difference in submission counting window I mentioned in the previous post. There is another issue that silently affected your team, due to an unfortunate mistake on our end.

The issue lies in the key debug: true in aicrowd.json that yoir team seems to have in all your submissions. This key has long been deprecated, and we do not include it in our starter kits. Due to legacy reasons, it is still present in our evaluations codebase, but will only allow 1 submission per day if the key is set to true. The submission just runs normally on the same dataset and the same settings.

However, it seems the mention of this key has not been removed from our autogenerated documentation for the Create Submission page. I presume this is where you got the key and added it to aicrowd.json? Normally, people do not add this key, and this issue went undetected.

In your case, since your team has 4 members, this led to the misunderstanding that every team member gets 1 submission each. Instead, they were failing because the debug key was set.

This is also the case with submissions from @quickpepper947, he was not using the debug key throughout the competition. But added it as true for one of the submissions, and then set it to false in the next submission. These two settings created seperate counters for the submissions, 1 for the debug set to true and 5 for the debug set to false. This led to 6 submissions going scored in one day.

We sincerely apologize for the confusion caused due to the debug key not being properly deprecated and removed from all communication channels. We’ll fix this issue as soon as possible.

#15

Hello @dipam, thanks for the updated info and now we have a better understanding on the issue. Yes we directly used the provided doc from the very beginning and did not change that throughout the challenge.

Just to confirm: given that our submissions with “debug=True” were successfully scored and updated on the Leaderboards, does it mean that all teams actually had 6 submissions per day as you mentioned (1 with “debug=True” and 5 with “debug=False”)?

Anyway hopefully this could be fixed ASAP to minimize its impact on other ongoing and upcoming challenges.

#16

@XavierJ , yes, all teams had the possibility to use 6 submissions due to this issue. However, to my best of my knowledge no one has used it intentionally. I apologize again that your team ended up having fewer submissions due to this.

I also checked @quickpepper947’s extra submission and 2 out of the 6 made on 8th May are actually identical commits.

#17

We also investigated our submissions again, I have two more findings

  1. we seemed to only have tested debug=True once on Mar 30th and left it to debug=False since then. We didn’t notice it back then but we can confirm that we got 6 submissions through on that day. As I remember, the reasons for enabling it, was to check weather we get more accurate readings on the runtime duration (not the case :slight_smile: ) Here is the blame log for aicrowd.json.

    image
    image2392×1132 335 KB

  2. The listing that @JusperLee posted above, showing that we got 7 submissions on May 8th UTC. seems not related to the debug issue. In fact, there seem to be two other reasons for it:

update: @dipam already mentioned the second issue in his post above

#18

@quickpepper947 We do not check for identical commits when counting submissions. I missed that the date you transitioned from debug false to true was not the date you got 6 submissions. In that case I’ll need to further dig into why you got 6 submissions.

I suspect that it may be related to the tags having identical timestamps as you mentioned, I guess further investigation is required.