Hi there,
I’m wondering a little about the reasons for the choice of submission format and evaluation process - will this be made clearer in subsequent rounds of this competition?
A much simpler method for evaluation would have been for everyone to submit a (num of molecules, total number of scent labels = 109) sized matrix (of 1 for a prediction of a scent label, 0 otherwise) and the just compute an overall Jaccard by computing:
Jaccard = TP/(FP+TP+FN)
This would be a fairer way to deal with molecules which varying label set cardinalities.
Regards,
Shabnam