🧞 Requesting feedback and suggestions

shivam · June 2, 2021, 8:01am

Hi all,

We are constantly trying to make this competition better for everyone and would really appreciate your feedback.

Feel free to reply to this thread with your suggestions and feedback on making the competition better for you!

What have been your major pain points so far?
What would you like to see improved?

Cheers!

In case you missed it, please make sure you set external dataset used properly in your aicrowd.json.

woosung_choi · June 2, 2021, 9:44am

Hi @shivam,

Many participants have struggled due to the time-out problem.
My team also got frustrated when we encountered inference failed at 100%.
Since some submissions failed at 100% inferences, I think the last track is the longest one.
So why don’t you add an additional phase for filtering time-out submission out with the longest track?
Then, participants will not have to wait for the entire inferences.
It will also reduce the evaluation system’s workload because it does not have to process all the tracks for time-out submissions.

I always appreciate your support.

Best,
Woosung Choi.

shivam · June 2, 2021, 10:07am

Hi @woosung_choi,

Thanks. Completely agree with your suggestion.

We will add one longer song in the validation phase itself for fail-fast asap. Plus, it will provide logs to everyone for debugging purposes as well.

woosung_choi · June 2, 2021, 10:25am

Thank you very much!

agent · June 2, 2021, 1:49pm

It will be great if organizers reproduce training of winning models from Leaderboard A at the end of the competition. Otherwise, participants can hide usage of extra data.

agent · June 2, 2021, 5:52pm

Will this be the longest song in the entire test dataset (28 songs)?

shivam · June 2, 2021, 5:56pm

Reminder Validation phase songs don’t count toward your leaderboard scores.

Tentative:
We wouldn’t release an additional song from private songs in the validation phase.
But will include an additional song from MUSDB18, etc having length ~same as the longest song in private songs.

shivam · June 4, 2021, 6:48am

Hi everyone,

We have added an extra song during the validation phase with a 03:30 min length.
Your submission runs for this song BUT the separated source isn’t counted towards any of the scores.

Timeouts, if any, should be visible early enough now on.

woosung_choi · June 4, 2021, 1:29pm

Good news! Thank you for your hard working!

agent · June 26, 2021, 10:06am

Participants are not able to remove their submissions from leaderboard. It would be great if it becomes possible. This could be useful for participants who set the external_dataset_used flag incorrectly.

sevagh · June 28, 2021, 10:54am

Not sure if my thinking on Leaderboard A vs. Leaderboard B is correct, but should models from leaderboard A supersede models from leaderboard B?

Hypothetically, if say:

Leaderboard A:

Model 1, SDR = 10.0
Model 2, SDR = 9.0

Leaderboard B:

Model 3, SDR = 7.0
Model 4, SDR = 6.0

Because model 1 and 2 have a higher SDR, do they also automatically “win” leaderboard B?

Basically, I can see both scenarios make sense:

Option 1: leaderboard A is strictly “external_dataset_used=False”, leaderboard B is strictly “external_dataset_used=True”
Option 2: leaderboard A is strictly “external_dataset_used=False”, in leaderboard B “external_dataset_used=True” is allowed, but all leaderboard A models are also automatically eligibile

agent · June 30, 2021, 5:06pm

I think the second option is preferable. Current “Leaderboard B” should be replaced by current “Overall”.

StefanUhlich · June 30, 2021, 5:14pm

@agent @sevagh Yes, I agree with @agent - the second option should be used and leaderboard B should also include systems which did not use external datasets (they are allowed to use extra data but don’t have to).

From experience, systems that are limited to use MUSDB18 train will not perform as good as systems that are allowed to use more data. Hence, the top systems of leaderboard A will not appear on top of the list of leaderboard B.

agent · June 30, 2021, 5:46pm

Thanks @StefanUhlich

By the way, the rules clearly state:

There are two leaderboards – one for systems that were solely trained on the training part of MUSDB18HQ (“Leaderboard A”) and one for systems trained on any data (“Leaderboard B”).

Hence, Leaderboard A is subset of Leaderboard B

I hope @shivam hears us )

shivam · June 30, 2021, 6:05pm

Hi @sevagh @agent, thanks for bringing it up. And clarification from Stefan.

Leaderboard B now includes the submissions without external dataset usage as well.

sevagh · July 1, 2021, 3:34pm

Great stuff. And yes, realistically I’d expect extra data to win, but I’m glad for the clarification.

If somebody can achieve the absolute best SDR with only MUSDB, that’s extra impressive and deserves double gold medals

agent · July 11, 2021, 2:26am

Feature request: ability to disband a team if there is only one member.

shivam · July 12, 2021, 1:58am

Unfortunately the way we have current teams feature designed we wouldn’t be able to provide it as an option for this challenge.

We have added it as feature requests on our side and should be able to provide it in future challenges.

sevagh · July 27, 2021, 5:10pm

Will there be an “open-source reveal day” or something? Presumably after the competition deadline/July 31 where contestants should make their code public?

It could be a real party to have 3 months of hidden work come to light.