[Announcement] Leaderboard Computation

NeurIPS 2019 : Disentanglement Challenge
#1

[Update]
The leaderboard is now updated with the following process:

  • For all submitted models of the participant we evaluate the average rank across all metrics on the leaderboard
  • We select the model with the highest average rank across all metrics on the leaderboard and call it the selected model.
  • We then remove all the submissions of this participant from the leaderboard, calculate the rankings on the leaderboard without him and repeat the procedure for the n-1 participants.
  • We keep doing this till we have the list of best submissions
  • We then calculate the final ranks for each row on the leaderboard

More details will follow soon as an annoucement.

[End Update]

Dear participants,

The Leaderboard is now being computed as described in the rules of the competition, (instead of just sorting based on FactorVAE as was being done until now.

Now you position on the leaderboard is being computed by the following approach :

  • For all the metrics individually

    • Compute the rank of all the submissions
    • The rank of the submission is defined by the index of the said submission, if all the submissions are sorted in a descending order

  • Then for all the submissions :

    • compute the mean_rank of each submission across all the metrics

  • Generate the leaderboard by :

    • Sorting all the submissions by their mean_rank (in an ascending order), and grouping by the individual participants : meaning only the “best submission” of a participant shows up on the leaderboard.

You can very well check the ranks of the said submission across all the metrics, by going into the “submission details” page (which you can access by clicking on the View button), and checking all the variables of the form _rank.

If there are any questions, or confusions, please feel free to reach out to us on this thread, or you can also optionally reach out to me directly at mohanty@aicrowd.com .

Cheers,
Mohanty

2 Likes
There is a bug in the leaderboard
[Announcement] Start of Round-2
#2

@mohanty
I believe there is a flaw with the suggested ranking system. I sent you an email where I have highlighted the potential issue and a possible fix.

Thanks.

#3

I think rank of SUM is good

#4

@mohanty, I think the rank of “sum of all score” should not be included in the mean of rank, because some metrics might suppress the other metric and the sum of all score will only reflect the part of those highest metrics.

1 Like
#5

@mohany
I want to propose an alternative approach for your consideration:

You can normalize each metric by dividing its value by the maximum value of that metric achieved across all the participants (i.e. only one participant is likely to get 1.0 out of 1.0 for each metric).
Once the above is done, simply sum over all metrics of a submission and rank accordingly.

#6

Hey everyone!

Please check the update to the original post in this thread!

More details will be sent to all participants via email.

Thanks!

1 Like