# Phase 2 update: flopscope v0.12.3, deadline extension, and final submission nomination

ARC White-Box Estimation Challenge 2026
1

Hi everyone,

We have updated the evaluators to flopscope v0.12.3. Alongside this update, we have extended the Phase 2 submission deadline by 4 days to 21 October 2026, 23:59 UTC. Nomination Dealine for the Phase 2 Private Re-evaluations is extended to 22 October 2026, 23:59 UTC, so teams can review the updated results before finalising their choices.

TL;DR

  • Phase 2 submissions close 21 October 2026, 23:59 UTC.
  • Nominate up to two valid submissions per team for the Phase 2 Private Re-evaluations by 22 October 2026, 23:59 UTC.
  • Algorithmic-contribution write-ups are due 28 October 2026, 23:59 UTC.
  • Prize rankings will come from the Phase 2 Private Re-evaluations.

What changed

The latest releases fix billing and symmetry issues, including undercounted broadcast work and truncated-SVD costs. Affected estimators can now be charged more FLOPs and receive different scores, even with identical code. Details are in the 0.12.2 and 0.12.3 release notes.

All affected submissions are being re-evaluated. Thank you to everyone who reported issues and contributed fixes.

Pull the latest starter kit and run uv sync. If you use pip:

python -m pip install --upgrade "flopscope>=0.12.3" "whestbench>=0.16.2,<0.17.0"

Please reconsider your submission candidates with the updated flopscope release before nominating your final submissions.

Final submission nomination

On the Submissions page, switch to Your team and use Select to nominate your submissions for the Phase 2 Private Re-evaluations. You can change your nominations until the nomination deadline, one day after submissions close.

If your team nominates nothing, we will use its two best-scoring valid Phase 2 submissions on the public leaderboard.

Prize rankings will be determined exclusively by the Phase 2 Private Re-evaluations, on fresh MLPs drawn from the Phase 2 distribution with private seeds. Nominate the submissions you expect to generalise best.

We plan to use flopscope v0.12.3 for these evaluations. If final review identifies security-critical issues, we may use an updated release. Any such changes will be communicated to participants and applied consistently to all affected nominated submissions, with reruns where necessary.

We look forward to your feedback here, please open reproducible bugs on GitHub issue tracker, or email rules questions and security reports to arc-whestbench@aicrowd.com.

All the best!
Mohanty

2 Likes
2

[Phase 2 write-up] The fourth-order closure that works is not the true one (submission #335513)

Hello ARC team and participants,

This is our technical write-up for the Phase 2 Algorithmic Contribution Prize.

Submission ID: 335513 (score 3.202e-9, 100 networks, 0 failures).

Main finding: at width 1024, the factored third-cumulant chain is right in level (<1 %); the remaining error sits at fourth order and beyond, and the fourth-cumulant closure that works (regenerated, lambda * C_off) is not the true one — imposing the true level makes the estimate worse by +29 % on 24/24 networks, even with all learned correctors removed. The note also reports per-component ablations of the submission, negative results, and one of our own earlier claims that re-verification refuted.

Transparency: most of this work, including the re-verification and the novelty audit, was carried out by AI agents (Anthropic’s Claude Opus and OpenAI’s Codex) directed by me.

We wanted to post this write-up publicly on the forum so that it can also be considered for the Community Contribution Prizes, but the “New Topic” button is not available on our account. Could you either enable topic creation for us, or post the attached PDF publicly on our behalf? Thank you.

The full PDF is attached.

note.pdf (454.9 KB)

3

@LuDoe, we saw the same thing from the other side. The inherited κ4 constant λ in the V29 line is 2–5× the value measured on Monte-Carlo truth, yet setting it to the measured value made our final MSE about 70 % worse in the shipped build, and about 40 % worse with the other fitted parts off (ablation, §5 of 18282). Part of the accuracy lives in error cancellation, which matches your title.

4

Thank you, Megan, a useful cross-check. Our comparable numbers (#335513, 24 public networks): replacing the fitted λ·C_off closure by the measured scale mode at its true per-layer level (65,536 samples per network) raised final MSE by +42.1 ± 15.5 % with our learned correctors, +28.8 ± 11.8 % without (24/24 worse). λ off (×1e-4) cost +85.8 % / +43.9 %; your λ = 0 arm on D gives ×1.74.

Converges: in this lineage, a truer κ4 makes the chain worse, and not only via learned parts. Differs: you change only the scalar λ (2.2e-3, one network); we change form and level, per network. We both inherit 504aldo’s λ table, so this is not independent evidence. Nor can we confirm cancellation: our own per-layer cancellation figure failed re-measurement.

Question: on which κ4 slice was the physical λ fitted? On earlier chains our fitted (3,1) slice sat near a third of the truth.

5

A third data point on the same question, with the same caveat: our law is also 504aldo’s one-scalar form, so this is not independent evidence.

  1. In our chain a truer fourth-cumulant core also made things worse. Carrying the core with transport and regeneration terms explained 92-94% of the exact core’s off-diagonal (R^2) at n = 256, yet gave x1.042 [0.997, 1.090] final MSE out of sample at n = 1024. With the reference’s own K211 production of the core in place of the law, the chain was about 7x worse (section 8 of our write-up, topic 18284).

  2. Our reading, not tested beyond these ratios, is that much of what the fitted lambda absorbs is il.miscusi’s scale mode (18248). With the mode taken out of the covariance, the third and the fourth cumulant each layer, the law can go to zero: final MSE x0.906 of the shipped law on mini 0-9 (10 of 10 lower), while the law at zero without the extraction costs x1.99 (section 7.6). That swap is the change in our submission 337488. With the extraction in place, carrying the core on top adds about 2% at best (x0.889).

  3. On how the lambda was fitted, for our line: it starts from the per-layer least-squares coefficient of the exact core’s off-diagonal on C_off along the exact reference chain at n = 256, scaled by 0.25 for n = 1024 (the 1/n power counting; a final-MSE scan put the best scale at 0.236-0.248). Five layer-group scales for layers 0-14 were then refitted on the final MSE of three networks, and all went up (+7% to +46%). So our shipped lambda sits above that scaled regression value, the same direction as both of yours, though by less.

6

Thanks — this fills a gap in our account. Our scale-mode arm swapped the mode in for the κ4 closure only, leaving the covariance and κ3 as they were: +297 % final MSE with our correctors, +756 % without, against the regenerated law. Your extraction from the covariance, κ3 and κ4 together, letting λ go to zero at ×0.906, suggests the mode has to be carried consistently across orders rather than patched into κ4 alone, which would explain both our failure and why the fitted λ sits high. We did not test the joint extraction, so we read this as your result, not ours. Thank you also for the fitting details.

7

Our partial versions point the same way. In the readable engine, against the f = 0.1, law x0.1 row, taking the mode out of the covariance alone gave x35 final MSE, out of the covariance and κ4 x25, and out of the covariance and κ3 x6.5 (section 7.6). We did not run κ4 alone. In the production engine with the law at zero, switching off one part at a time on mini 0-7, leaving out the core-row, covariance or new-leg part gave 5 to 28 times shot 38’s final MSE with the band on, while leaving out the feed pair stayed near the full version (section 5). We read these only as the parts not working apart in our implementation, not as separate contributions of each order.