Thanks a lot for participating in stage 1 of the NeurIPS 2019 Disentanglement Challenge. In addition to the already released simple simulations (mpi3d_simple), we now release the data from the public leaderboard of the first stage i.e. the realistic simulations (mpi3d_realistic).
In stage 2, we will now try to advance the unsupervised learning of disentangled representations to more difficult objects. In stage 2 both the public and private leaderboard will consist of real-world images.
In stage 2 your are allowed to use all the released images of stage 1 and we encourage you to have a look and review the submitted reports on openreview from stage 1 as well.
The timelines are as follows:
Sept 2nd 20:00 CET: Start of Stage 2 of the NeurIPS 2019 Disentanglement Challenge
Oct 1st: 11:59pm AoE Submission deadline for methods, Stage 2
Oct 7th: 11:59pm AoE : Submission deadline for reports, Stage 2
Is the ranking algorithm as was discussed: average of the rank of each metric for each participant?
If so, I guess I missed by 1 rank to be among the top 3
We select the model with the highest average rank across all metrics on the leaderboard and call it the selected model.
I recall we were asked to nominate one of our submissions as the one we assume to be the best. I wonder which path the committee ended up taking: choosing the best model automatically, or based on the user’s selection.
@amirabdi: The current computation actually chooses the best submission for you automatically. I havent closely looked at how well they agree with the nominated ones, but if they do not match, your rank would only go down
