Which submission is used for private LB scoring?

Data Purchasing Challenge 2022
,
#1

I couldn’t find it anywhere stating this. Which of the following statements is true?

  1. all submissions to round 2 will be evaluated on private LB and the best score is picked automatically for each participant.
  2. only the best submission on public LB of each participant will be selected for private LB scoring
  3. each participant needs to specify which submission to use for private LB scoring.

Thank you in advance for clarification!

2 Likes
#2

good question!
If it would be 1., there would be an incentive to run many variations covering many potential distributions in the hope of one fitting best. So this seems bad.
2. seems plausible. But it has the danger that this submission is overfitted to the public leader board. It would incentivize not trying out many submissions.
3. seems best, but there is no feature currently where you can specify this.

2 Likes
#3

Hi @xiaozhou_wang, @tfriedel,

Thanks for the question, and discussion here.

We will be allowing participants to select up to 3 submissions of their choice for the private leaderboard.
A form will be floated at the end of the Round-2 for the same.

3 Likes