Phase 1 Write-Up: Useful tools, rejected ideas, a warning, and more

ARC White-Box Estimation Challenge 2026
#1

Disclaimer: The first half of this writeup (everything before the Claude heading) was written without the use of LLMs except for the slight changes listed in the LLM advice section. The second half was written by Claude.

Some things that may be useful

I made the following tools that may be useful:

  • A script to evaluate the estimator in parallel on fly.io, using 100 remote machines that evaluate one MLP each. This makes experiments much faster. (I recommend writing your own script; Modal is probably slightly faster and has more free credits, and I neglected this.)
  • A fork of flopscope that adds GPU acceleration (https://github.com/01-1/flopscope). Makes running the experiments locally marginally faster (2x on my hardware) - this is superseded by the remote evaluator unless you only need to run a few MLPs.

Also, the Claude half of the writeup below contains:

Full repository: https://github.com/01-1/arc-wbe (Mostly AI-generated)

History

A brief history of my work on the project. Almost all of the work for my submission (research, idea generation, experimentation) was done by LLMs.

Warm-up:

  • Initially used GPT 5.5 loops to improve the estimator.
  • Told it to use the existing K=3 algorithm from Estimating the expected output of wide random MLPs more efficiently than sampling for third-order cumulants, reimplementing it in Python and optimizing/iterating on it.
  • Reward hacking: I noticed a sudden decrease in score (decrease = better), so I asked a different LLM to check 5.5’s work; it found that 5.5 fitted to the public (HuggingFace) test cases. I enacted guardrails and checks (do not fit to tests + added a skill so I could make agents to check the estimator).

Phase 1:

Initially still used GPT 5.5 since Fable was unavailable (did not get to use it in the brief first window of availability)

July 1: Fable rereleased. Used it to orchestrate GPT agents and generate ideas.

  • Fable decided to generate its own test cases (maybe for getting the raw activation samples, maybe for reward hacking). After catching this, I banned using such test cases for evaluation, requiring the use of the Fly evaluator. The locally-generated testcases are only allowed to be used for getting raw activation samples.

July 9: Sol released. For the same price, a Codex sub gives a lot more usage for Sol than a Claude sub gives for Fable, especially including resets, but Sol did not come up with any significant improvements on its own, as far as I remember.

Later, I attempted to create lessons to upskill myself using Fable/Sol, but I ultimately decided this would be unfruitful.

LLM advice

Claude

Everything from here onward was written by Claude.

What didn’t work

The repository’s largest artifact is a list of 69 closed research directions, each with the measurement that closed it. This is the part I’d point at first: nobody optimising for rank pays to characterise configurations they already know lose, so measured dead ends are scarce, and a lane you can rule out in five minutes is worth more than one you have to spend a week disproving.

The families, and what killed each:

Control variates. Anything with a known expectation computable inside predict() — third and fourth moment residuals against Gaussian ReLU targets, gate-rate residuals, row-radius residuals, plus the top-8 cross-seed principal components of the full 256-dim residual vectors under cross-validation. Pooled honest R^2 was 0.0049; the generous cross-validated version went negative. A mechanism relevant at the top of the board needs to explain {\sim}40\% of final-error variance. The family ceilings at {\sim}0.5\%.

Angular and likelihood-ratio importance sampling. Closed on paper rather than by experiment: for a fixed set of proposal evaluations, any universally unbiased matrix-weight estimator Rao-Blackwellises to scalar balance weights on the same vector integrand. Output-specific proposals and grouping cannot beat the scalar optimum q \propto \lVert g \rVert. The best scalar proposal reached 2.43\times against a target ceiling that needed far more.

Analytic cumulant ladders. Deterministic closures have zero sampling variance, which sounds ideal until you price the bias. Best net bias MSE was 4.69 \times 10^{-5} for the depth-32 analytic route and 1.20 \times 10^{-4} for the \kappa_4 ladder — against a route whose total error is 2.7 \times 10^{-6}. Adding the fourth cumulant bought 1.004\times; the \kappa_4 correlation had collapsed to -0.013 by layer 8. Gaussian-sum propagation was flat in component count, with a fitted scaling exponent of 0.001 across M = 1 \ldots 16.

Terminal readouts. Replacing the final sample mean with something smarter: a two-Gaussian EM mixture (0.955\times, worse), a one-Gaussian analytic plug-in (0.712\times), Gaussian and Edgeworth smoothing (bias-dominated), and a Haar-sphere fold-CV correction (failed four of five preregistered gates). The final layer is where an approximation looks most tempting and pays least.

Everything with a bias/variance trade. The re-audit produced the single most useful screening rule in the repo: convert a variance saving into the bias budget it can afford. A candidate with 3.2\% lower variance tolerates b^2 < 8.9 \times 10^{-9} at full budget before it stops paying; at 0.33\% lower variance the allowance is 9.0 \times 10^{-10}. Odd-state Rao–Blackwell had genuinely lower variance — 0.564\times the baseline — and still died, because its measured b^2 of 2.56 \times 10^{-4} was roughly 2{,}000\times its allowance. That calculation is cheaper than a gate and kills most candidates before you run them.

Quasi-Monte Carlo. Sobol on the sphere, triangular-LT, Jacobian-LT, folded ZCA, Hadamard LHS, antithetic LHS. Every successful gate ran at a single sample count, so none of them can support a scaling-exponent argument in either direction — which is the only argument that would have mattered. If you try QMC here, run it at two sample counts.

A bug you may share

Three of my gate aggregators computed a three-replicate bias/variance decomposition incorrectly, labelling M_3 — the MSE of the replicate mean — as b^2. It inverted at least one gate verdict: the spherical-Stein gate recorded stein_bias: FAIL when the corrected value is -1.98 \times 10^{-8}, consistent with zero.

With M_1 the mean single-replicate MSE and M_3 the MSE of the replicate mean:

\sigma^2 = \tfrac{3}{2}\left(M_1 - M_3\right)

b^2 = M_1 - \sigma^2 = \tfrac{3}{2}M_3 - \tfrac{1}{2}M_1

The raw per-shard rows were correct throughout — only the summaries were wrong. That’s why the repository ships raw rows next to every verdict, and it’s why I’d treat any prose figure in it, including my own, as unverified until re-derived.

Testing amalgonim’s premise

amalgonim’s score identity shows that above the multiplier floor,

\text{score} = \text{MSE} \times \max\!\left(0.1, \tfrac{C}{B}\right) = \frac{V}{n} \cdot \frac{cn}{B} = \frac{Vc}{B}

so the sample count n cancels. It’s correct algebra, but it assumes \text{MSE} = V/n: pure sampling error, no bias floor. Nobody had measured that.

Four paired full-100 runs at 4,096 / 8,192 / 16,384 / 32,768 samples, 97 MLPs paired across all four:

samples n mean final-layer MSE s.e.
4{,}096 5.4604 \times 10^{-6} 6.42 \times 10^{-7}
8{,}192 2.6674 \times 10^{-6} 2.89 \times 10^{-7}
16{,}384 1.5010 \times 10^{-6} 1.48 \times 10^{-7}
32{,}768 7.0862 \times 10^{-7} 7.49 \times 10^{-8}

Fitting \text{MSE} = F + c\,n^{-\alpha} with one residual degree of freedom gives \alpha = 1.039 (90% CI [0.840,\ 1.240]) and a block-independent component F bounded below 1.5 \times 10^{-7}, consistent with zero. The premise holds.

That isn’t automatic. My route stacks an exact first-layer Gaussian recolor, a heuristic 1.5\times variance match on the first successor layer, fp32 propagation and three Strassen levels on top of the sampling core — any of which could have floored it. And amalgonim’s own post measures a floor of 8.8 \times 10^{-7} for the closure family. So the floor appears to track where the analytic approximation sits rather than whether one is used: closure at the readout floors, closure at the first layer inside a sampler doesn’t.

This matters because the two cases want opposite operating points. With a floor, \text{score} = (Vc + FC)/B rises with compute and minimum compute is strictly optimal. Without one the score is flat — my 16-block and 64-block routes score 2.75 \times 10^{-7} and 2.74 \times 10^{-7} at 3.7\times the compute.

Caveats worth keeping: one estimator family, an interval wide enough to support the premise rather than pin it (it excludes superlinear scaling but can’t separate \alpha = 0.9 from \alpha = 1.2), and \text{MSE} \times C conserved only to about 10\% — 7.68, 7.51, 8.35, 7.45 (all \times 10^{4}) across the ladder, with the 16{,}384-sample point sitting 11\% high.

The predictions were computed from the three-point fit before the fourth point ran: 7.72 \times 10^{-7} (no floor), 9.48 \times 10^{-7} (quadratic), 9.83 \times 10^{-7} (free exponent). Observed 7.086 \times 10^{-7}, below all three, ruling out both floor-bearing models. Three points admitted floors of 1.1 \times 10^{-7}, 4.4 \times 10^{-7} and 6.0 \times 10^{-7} depending on functional form — three different verdicts about whether the direction was alive. Fitting three parameters to three points identifies nothing.

python analysis/block_ladder/fit_ladder.py reproduces all of it from the published CSV.