Questions about leaderboards A and B

Music Demixing Challenge ISMIR 2021
4

Dear Dr. @StefanUhlich and @shivam
I have similar questions related to this issue.

  1. Can we tune hyper-parameters by analyzing the SDRs measured by AICrowd? For example, what if we choose the best checkpoint after submitting top-3 checkpoints? Should it be submitted to A or B? Or is this not allowed?

  2. Challenge Rules said, “one for systems that were solely trained on the training part of MUSDB18HQ.”
    However, there are two versions of splits in musdb18: (84 training, 16 validation, 50 test) and (100 training, 50 test) as far as we know. I am not sure this means whether we should use the predefined (86 training, 14 validation, 50 test) split for leaderboard A or not. Can we use customized splits in 100 training tracks to submit a model to leaderboard A? - for example, 95 training items and five validation items?

Best regards,
Woosung Choi.

3 Likes