I looked at the rules and am just wondering how prizes are decided in phase 1.
Are the 50 public MLPs included in final evaluation in phase 1?
In that case are we allowed to make solutions that overfit to them?
For example I notice this submission: AIcrowd | ARC White-Box Estimation Challenge 2026 | Submissions #321819
suspiciously has different behavior on the public MLPs vs. the private ones. (amount of flops used) in a way that is very improbable by random chance: all the private ones use the exact same amount of flops while the public ones vary.
@SKIBIDI_TOILET All Phase 1 submissions are evaluated on 50 public MLPs and 50 private MLPs. The private set allows us to monitor how closely public and private performance track, and to identify potential overfitting to the public instances.
At the end of Phase 1, each team may select up to two submissions for the Phase 1 private re-evaluation. If a team does not explicitly specify its choices, we will use its two highest-ranked submissions on the Phase 1 public leaderboard.
For the private re-evaluation - conducted after Phase 1 and again at the end of Phase 2 - we will re-run each eligible participant’s selected submissions on a completely separate, freshly generated test suite of randomly initialized MLPs. These MLPs will be generated using private seeds that were not used during either Phase 1 or Phase 2.
The number of MLPs included in each private re-evaluation will be determined based on the final set of eligible submissions. We will use appropriate statistical analyses to select a test-suite size that allows us to reliably distinguish between closely performing solutions. The exact number of MLPs may therefore vary between re-evaluations.
This re-run will produce the final private leaderboard, and prize rankings will be determined exclusively by these private re-evaluation results - not by any score displayed on the public leaderboard during the competition.
The private re-evaluation suite will be drawn from the same distribution as the corresponding phase suite, including the same width and layer-count ranges and the same FLOP-budget calibration. However, it will contain entirely different MLP instances. As a result, solutions that overfit to specific public MLPs or seeds are unlikely to generalize well.
Therefore, teams that deliberately optimize for the public leaderboard should expect this to be reflected both in their Phase 1 public–private score divergence and in the post-phase private re-evaluations.
2 Likes
Cool, thank you for the clarification.