[Announcement] Spleeter Baseline should have external dataset "true"

Spleeter baseline has 2nd highest score and many of us might be using it. :tada:

But we noticed that participants who are using it missed setting the external_dataset_used flag to true.

Please set it to true for new submissions, and you can share older submission IDs to me via [:e-mail:Message] or reply to this thread. Happy participating! :angel:

2 Likes

Thank you very much for your hard-working Shivam!
It is just a personal opinion, but I think there are still many people who submit models trained with extra datasets to leaderboard A.

Also, models based on pre-trained models such as demux-extra (https://github.com/facebookresearch/demucs/blob/2404fa029137b39d66a46db65958cffd50bed6ad/demucs/pretrained.py#L23) and tasnet-extra (https://github.com/facebookresearch/demucs/blob/2404fa029137b39d66a46db65958cffd50bed6ad/demucs/pretrained.py#L26) must be placed in leaderboard B.

2 Likes

Hi @woosung_choi, yes, I agree that there are still a few submissions on wrong leaderboard.

The flow depends on self-declaration and participants can end up marking external_dataset_used as false instead of true by mistake.
We are reaching out to participants so they can provide us submission IDs for older submissions with the wrong tag, and for having the correct tag in newer submissions.

And +1 for demux-extra thing, it belongs to Leaderboard B.

3 Likes