I looked at the rules and am just wondering how prizes are decided in phase 1.
Are the 50 public MLPs included in final evaluation in phase 1?
In that case are we allowed to make solutions that overfit to them?
For example I notice this submission: AIcrowd | ARC White-Box Estimation Challenge 2026 | Submissions #321819
suspiciously has different behavior on the public MLPs vs. the private ones. (amount of flops used) in a way that is very improbable by random chance: all the private ones use the exact same amount of flops while the public ones vary.
1 Like
@marius_binner All Phase 1 submissions are evaluated on 50 public MLPs and 50 private MLPs. The private set allows us to monitor how closely public and private performance track, and to identify potential overfitting to the public instances.
At the end of Phase 1, each team may select up to two submissions for the Phase 1 private re-evaluation. If a team does not explicitly specify its choices, we will use its two highest-ranked submissions on the Phase 1 public leaderboard.
For the private re-evaluation - conducted after Phase 1 and again at the end of Phase 2 - we will re-run each eligible participant’s selected submissions on a completely separate, freshly generated test suite of randomly initialized MLPs. These MLPs will be generated using private seeds that were not used during either Phase 1 or Phase 2.
The number of MLPs included in each private re-evaluation will be determined based on the final set of eligible submissions. We will use appropriate statistical analyses to select a test-suite size that allows us to reliably distinguish between closely performing solutions. The exact number of MLPs may therefore vary between re-evaluations.
This re-run will produce the final private leaderboard, and prize rankings will be determined exclusively by these private re-evaluation results - not by any score displayed on the public leaderboard during the competition.
The private re-evaluation suite will be drawn from the same distribution as the corresponding phase suite, including the same width and layer-count ranges and the same FLOP-budget calibration. However, it will contain entirely different MLP instances. As a result, solutions that overfit to specific public MLPs or seeds are unlikely to generalize well.
Therefore, teams that deliberately optimize for the public leaderboard should expect this to be reflected both in their Phase 1 public–private score divergence and in the post-phase private re-evaluations.
5 Likes
Cool, thank you for the clarification.
@mohanty How will the teams be able to select the 2 submissions for the phase 1 private evaluation? Will there be some UI for that or will we receive an email requesting the id of the 2 submissions?
@mohanty Following up on @ngcferreira’s question above, now for Phase 2: how should teams specify their (up to) two submissions for the Phase 2 private re-evaluation? Is there a selection form, or should we email arc-whestbench@aicrowd.com with the submission ids before the phase closes on 17 October? And if a team specifies nothing, are its two highest public leaderboard submissions used, as in Phase 1?
A related question: will the private re-evaluation bill FLOPs with the same flopscope version as the public board (0.12.1), or with a version that includes fixes for the billing issues reported on GitHub (AIcrowd/flopscope issues #264 to #278)? Knowing this before the deadline would let teams check their selected submissions against it. Thank you.
@ngcferreira @AndreasHad04 : We will enable the submission selection 1 week before the Phase 2 final submission deadlines, and we will make an announcement on the forums and email to all the participants.
Before that, we plan to address some of the flopscope disclosures we received from participants, release a new version of flopscope, re-evaluate all the affected submissions, and only then enable the submission selection for the post-phase Private re-evaluations.