Which tag in git will you choose for final ranking?

NeurIPS 2019 : Disentanglement Challenge
#1

Hi,

I have 11 submition (11 tags), Which tag in git will you choose for final ranking???

Thanks

#2

The final rankings will be computed by computing the rank of each participant across all the individual metrics. And then taking the mean of the rank of the participant across all the metrics. Across all the valid submissions you have made.

1 Like
#3

Can you clarify this further please? What exactly is the rank? Does the best score on a metric get rank 1, second best score gets rank 2, etc?

Furthermore, do you average over all submissions, or do you just take the submission with the best scores into account?

2 Likes
#4

By rank here, what I meant was :

  • For a single metric :
  • take the best submission of each participant
  • Sort all these submissions
  • rank of a participant is defined as the index of the participant in this sorted list
  • Repeat the above for all the metrics
  • Compute the mean of the participantā€™s rank across all the metrics, and assign it as say mean-R for each participant.
  • Now sort the list of participants based on the mean-R values, and then the order generated (lower the better), is the final leaderboard.
1 Like
#5

Hello, I have a question. The best submission means the highest FactorVAE score or the best mean rank?

#6

Can you show the final rank as soon as possible? Because the rank based on FactorVAE is puzzlingļ¼ŒThank you very much!!

#7

@jozerozero : The leaderboard has a messgae on top at the moment, which says that the current rankings are tentative. And the approach described above will indeed be used to compute the final rankings.
The tentative leaderboard is computed using FactorVAE, but we will soon release the actual leaderboard based on the mean-ranks as described above.
Thank you for your patience.

Cheers,
Mohanty

1 Like
#8

Hi. @mohanty So, for each metric, what is the best submission? Does it mean that for each metric, it choose the different submissions?

2 Likes