Hi everyone,
We have updated the evaluators to flopscope v0.12.3. Alongside this update, we have extended the Phase 2 submission deadline by 4 days to 21 October 2026, 23:59 UTC. Nomination Dealine for the Phase 2 Private Re-evaluations is extended to 22 October 2026, 23:59 UTC, so teams can review the updated results before finalising their choices.
TL;DR
- Phase 2 submissions close 21 October 2026, 23:59 UTC.
- Nominate up to two valid submissions per team for the Phase 2 Private Re-evaluations by 22 October 2026, 23:59 UTC.
- Algorithmic-contribution write-ups are due 28 October 2026, 23:59 UTC.
- Prize rankings will come from the Phase 2 Private Re-evaluations.
What changed
The latest releases fix billing and symmetry issues, including undercounted broadcast work and truncated-SVD costs. Affected estimators can now be charged more FLOPs and receive different scores, even with identical code. Details are in the 0.12.2 and 0.12.3 release notes.
All affected submissions are being re-evaluated. Thank you to everyone who reported issues and contributed fixes.
Pull the latest starter kit and run uv sync. If you use pip:
python -m pip install --upgrade "flopscope>=0.12.3" "whestbench>=0.16.2,<0.17.0"
Please reconsider your submission candidates with the updated flopscope release before nominating your final submissions.
Final submission nomination
On the Submissions page, switch to Your team and use Select to nominate your submissions for the Phase 2 Private Re-evaluations. You can change your nominations until the nomination deadline, one day after submissions close.
If your team nominates nothing, we will use its two best-scoring valid Phase 2 submissions on the public leaderboard.
Prize rankings will be determined exclusively by the Phase 2 Private Re-evaluations, on fresh MLPs drawn from the Phase 2 distribution with private seeds. Nominate the submissions you expect to generalise best.
We plan to use flopscope v0.12.3 for these evaluations. If final review identifies security-critical issues, we may use an updated release. Any such changes will be communicated to participants and applied consistently to all affected nominated submissions, with reruns where necessary.
We look forward to your feedback here, please open reproducible bugs on GitHub issue tracker, or email rules questions and security reports to arc-whestbench@aicrowd.com.
All the best!
Mohanty