Hi @dipam , just need a little clarifications about your post :
The detailed steps are given below:
Eligible teams will select two of their submissions to evaluate - Eligibility criteria to be announced soon, it will be based on Round 2 leaderboard.
Each submission will run through the pre-train and the purchase phase on the end of competition dataset.
The same purchased labels will be put through 5 training pipelines - Details to be released soon.
Each training pipeline will be run for 2 seeds and scores averaged, to address any stochasticity in scores.
To avoid issues due to difference of average scores from different training pipelines, a Borda ranking system will be used.
while the 5 training pipelines results scored using Borda ranking system, how about the submissions? is it the highest score from the submission that is being used or is it an average from both submission results?
No, all the runs are completed. The leaderboard is final. The best scoring submission of each participant from the full runs was used. The “successful entries” column is generic feature of the leaderboard, since leaderboard was calculated offline and the final results added separately, it shows 1.
Please check this sheet with the full scores of all the runs.