What might cause a "Scoring failed"?

Music Demixing Challenge ISMIR 2021
#1

hi, I’m wondering what might cause a “Scoring fail” error. My recent submission failed with this error and I’m a little confused.

2 Likes
#2

I’ve been getting the “scoring failed” error as well. All my submissions have failed with that same error since they added the extra song for early timeout checking.

3 Likes
#3

So “scoring failed” here means “inference time out”? I have checked the log. The 3:30 long song takes 220s to separate. It seems the problem is not about time out.

2 Likes
#4

Yeah I noticed too. My submissions also doesn’t exceed 240s so i really don’t know what the problem is.

2 Likes
#5

I guess that the updated validation phase recently added might have a too strict threshold.
It seems that the models, which have passed the entire evaluation phases before, are currently filtered out after the update.
Hi @shivam, sorry to bother you gain, but could you please let me know the threshold used for time-out submission filtering?

2 Likes
#6

Hi @HaoheLiu @kim_min_seok @woosung_choi,

Thanks for notifying us about the error.
And yes, it indeed happened after adding the additional song for time-out filtering. :cry:

We have fixed the issue now and all the submissions which failed due to it are reevaluated.
In case your submission isn’t reevaluating already, please let me know the submission ID by :email: direct message.

Happy participation!

3 Likes