🚨 Select submissions for final evaluation

Hi everyone,

Thanks for participating in the Data Purchasing Challenge! :rocket:

We hope you had fun with the problem and the dataset. In this next phase, we will be evaluating your selected submissions on the full hidden dataset.

‎ℹ️ Know more about it here:

:alarm_clock: Deadline for filling this form is 08 April 2022, 23:59 UTC (earlier the better, thanks :dancer:)

:point_right: Select Submission ID for Full Evaluation

3 Likes

Hi @dipam , just need a little clarifications about your post :

The detailed steps are given below:

  1. Eligible teams will select two of their submissions to evaluate - Eligibility criteria to be announced soon, it will be based on Round 2 leaderboard.
  2. Each submission will run through the pre-train and the purchase phase on the end of competition dataset.
  3. The same purchased labels will be put through 5 training pipelines - Details to be released soon.
  4. Each training pipeline will be run for 2 seeds and scores averaged, to address any stochasticity in scores.
  5. To avoid issues due to difference of average scores from different training pipelines, a Borda ranking system will be used.

while the 5 training pipelines results scored using Borda ranking system, how about the submissions? is it the highest score from the submission that is being used or is it an average from both submission results?

Hi @leocd

The highest scoring among the selected submissions will be used.

2 Likes

Hi @dipam, thanks for hosting the interesting compeitition!
It seems the competition was finished, when will the leaderborad be finalized?

1 Like

Hi, we are working on the final evaluations and you can expect the results by 14th April.

Meanwhile the submissions selection is still open and you can select your submissions [if not done already] here:
https://www.aicrowd.com/challenges/data-purchasing-challenge-2022/dynamic_contents

1 Like

Hi @shivam, is there any progress?

1 Like

Hi @shivam @dipam, do you have any timeline for the leaderboard update?

2 Likes

hi @shivam it seems that evryone only has 1 successful entry, does it mean that it will take another 7 days to have the final result?

Hi @chuifeng

No, all the runs are completed. The leaderboard is final. The best scoring submission of each participant from the full runs was used. The “successful entries” column is generic feature of the leaderboard, since leaderboard was calculated offline and the final results added separately, it shows 1.

Please check this sheet with the full scores of all the runs.

Hello, You can find the final leaderboard and top three ranking winners over here.