🚨 Select submissions for final evaluation

Data Purchasing Challenge 2022
#1

Hi everyone,

Thanks for participating in the Data Purchasing Challenge! :rocket:

We hope you had fun with the problem and the dataset. In this next phase, we will be evaluating your selected submissions on the full hidden dataset.

‎ℹ️ Know more about it here:

:alarm_clock: Deadline for filling this form is 08 April 2022, 23:59 UTC (earlier the better, thanks :dancer:)

:point_right: Select Submission ID for Full Evaluation

3 Likes
#2

Hi @dipam , just need a little clarifications about your post :

The detailed steps are given below:

  1. Eligible teams will select two of their submissions to evaluate - Eligibility criteria to be announced soon, it will be based on Round 2 leaderboard.
  2. Each submission will run through the pre-train and the purchase phase on the end of competition dataset.
  3. The same purchased labels will be put through 5 training pipelines - Details to be released soon.
  4. Each training pipeline will be run for 2 seeds and scores averaged, to address any stochasticity in scores.
  5. To avoid issues due to difference of average scores from different training pipelines, a Borda ranking system will be used.

while the 5 training pipelines results scored using Borda ranking system, how about the submissions? is it the highest score from the submission that is being used or is it an average from both submission results?

#3

Hi @leocd

The highest scoring among the selected submissions will be used.

2 Likes
#4

Hi @dipam, thanks for hosting the interesting compeitition!
It seems the competition was finished, when will the leaderborad be finalized?

1 Like
#5

Hi, we are working on the final evaluations and you can expect the results by 14th April.

Meanwhile the submissions selection is still open and you can select your submissions [if not done already] here:
https://www.aicrowd.com/challenges/data-purchasing-challenge-2022/dynamic_contents

1 Like
#6

Hi ＠shivam, is there any progress?

1 Like
#7

Hi @shivam @dipam, do you have any timeline for the leaderboard update?

2 Likes
#8

hi @shivam it seems that evryone only has 1 successful entry, does it mean that it will take another 7 days to have the final result?

#9

Hi @chuifeng

No, all the runs are completed. The leaderboard is final. The best scoring submission of each participant from the full runs was used. The “successful entries” column is generic feature of the leaderboard, since leaderboard was calculated offline and the final results added separately, it shows 1.

Please check this sheet with the full scores of all the runs.

#10

Hello, You can find the final leaderboard and top three ranking winners over here.