Are these wrong leaderboard submissions? Will they be removed?

Sound Demixing Challenge 2023
#1

@dipam @mohanty
The following submissions have the exact same score. The first three are on leaderboard A and B.

#218307, #219416, #218308 (currently 3rd ranked on Leaderboard B)

These next three are on leaderboard C.

#211429, #217973, #219408

Out of these one was posted as wrongly submitted to leaderboard A

So I am wondering if the other entries on leaderboards A and B with the same score from different participants are also on the wrong leaderboards. :thinking:

#2

Thanks for pointing this out, I’ll follow up with the participants.

Of course in general, we cannot prevent wrongly marked submissions from showing up on the leaderboard if the users don’t tell us to remove them.

However, for the data constrained leaderboards of both CDX (LB A) and MDX (LB A and B), we will collect the training code from the winners and reproduce the results, hence any wrongly marked submissions will certainly be removed in the final leaderboards.

2 Likes
#3

I just realized these are the exact same scores of the Baseline XUMX-M model for leaderboard C. They have the same Phase 1 scores. But, since the baseline was not run for Phase 2, it’s not immediately apparent.

#4

I can confirm the model (#219416) that I submitted is a baseline model. I was testing my code but forgot to change the aicrowd.json setting.

Not sure about other participants, but if they got exactly the same score as mine, then those should also be the baseline model.

#5

@subatomicseer , I inspected the inference code and indeed the submissions you mentioned were baseline models, not allowed for leaderboard B and C. Thank you for mentioning these.

1 Like