[Phase 1 write-up] Variance reduction beats closure refinement: a moment-matched sampling estimator (submission #324497)

ARC White-Box Estimation Challenge 2026
1

Variance reduction beats closure refinement: a moment-matched sampling estimator, and why our analytic estimator lost to naive Monte Carlo

Summary

We entered with a full-covariance analytic estimator and ended with a moment-matched antithetic sampler that scores 23x better. The interesting part is not the sampler, which is standard variance reduction. It is why the analytic route lost, and what the scoring rule actually rewards.

Three findings we think generalise:

  1. Our analytic estimator was accuracy-identical to the bundled 03_covariance_propagation.py example, to three significant figures on every metric. Three separate attempts to improve its ReLU off-diagonal — zero-mean arc-cosine (Cho-Saul), a second-order Price correction, and the theoretically exact Drezner-Wesolowsky bivariate orthant — all failed to beat the crude product gain, and the exact one was worst. The Gaussian-closure error and the product-gain error partially cancel, so refining the off-diagonal alone degrades the scored mean.
    1. Naive Monte Carlo beat that analytic estimator by 4.4x on the scored metric at n=5,000. Under this scoring rule, closure refinement was the wrong axis entirely.
      1. The compute multiplier makes sample count a corner solution, not a tuning knob. The optimum is the largest n whose utilization is still at the 10% floor, and both sides of that point are strictly worse for different reasons.

    3. The scoring rule determines the search space

  3. Score is adjusted = MSE x max(0.1, C_m / B) with B = 2.72e11 per MLP.
  • Below 10% utilization the multiplier is pinned at 0.1, so adjusted = MSE / 10 and MSE falls as 1/n. Every unused FLOP is wasted accuracy.
    • Above 10%, adjusted = MSE x C_m/B. With MSE ∝ 1/n and C_m ∝ n, the product is constant. The score is flat: extra samples buy nothing.
  • So the whole family of pure-sampling estimators collapses to a single number determined by variance per FLOP, and n only decides whether you have reached that number. We measured this rather than assumed it. Our n=5,000 submission graded at 7.97% utilization — under the floor, and therefore leaving accuracy unclaimed. Doubling to n=10,000 moved utilization to 15.93% and improved the graded score from 3.85e-7 to 2.88e-7. A third doubling would gain nothing.

This also means an estimator can be beaten by an identical one that merely spends its budget.

What we built

A Monte-Carlo layer-mean estimator whose sample set matches the target distribution’s first two moments exactly.

Antithetic pairing (first moment). Draw n/2 inputs and use each with its negation. Because N(0, I) is symmetric this is unbiased, and the empirical mean is exactly zero by construction. It pays because the first layer has a large odd component: ReLU(z) + ReLU(-z) = |z|, so the pair average has variance (1/4)(1 - 2/pi) sigma^2 ~= 0.091 sigma^2 against 0.170 sigma^2 for two independent ReLU draws — about 1.9x on layer 1, decaying with depth as activations become non-negative. Measured: 1.33x (1.17e-05 → 8.81e-06), consistent with that decay.

The pairing is also free at layer 1: ReLU(-z) = ReLU(z) - z, so one width x width matmul serves both branches.

Covariance whitening (second moment). Force the empirical covariance to exactly I. With G = X^T X / (n/2) = L L^T, taking X_w = X @ inv(L)^T gives X_w^T X_w = (n/2) I exactly, so the full antithetic set [X_w; -X_w] has both moments exact. Cost is O(n·width^2 + width^3) ~ 1.6e8 against ~4.3e10 for the forward passes, i.e. free. Measured: a further 2.1x on final-layer MSE (8.81e-06 → 4.18e-06) and 2.7x on all-layers MSE — the larger all-layers gain is the expected signature, since whitening helps most where the map is closest to quadratic, and early layers are.

Requires n/2 >= width for a full-rank Gram matrix; we guard and fall back to plain antithetic otherwise, since a Cholesky failure would zero every prediction for that MLP.

Results

All from whest run on hf://aicrowd/arc-whestbench-public-2026@v1-phase1 (100 MLPs, baked N=1e9 ground truth), or from AICrowd grading.

| estimator | submission | local MSE (100-MLP public split) | graded adjusted (AICrowd) |

|—|—|—|—|

| analytic covariance propagation (= bundled example) | #318255 | 8.37e-05 | 6.62e-06 |

| plain MC, n=5,000 | not submitted | 1.17e-05 | — |

| antithetic MC, n=5,000 | #324486 | 8.81e-06 | 8.01e-07 |

| moment-matched antithetic, n=5,000 | #324493 | 4.18e-06 | 3.85e-07 |

| moment-matched antithetic, n=10,000 | #324497 | 2.42e-06 | 2.88e-07 |

The two MSE columns are kept separate on purpose. The local column is a controlled ablation — same 100 MLPs, one variable changed at a time — and the ablation ratios quoted above come from it. The graded column is AICrowd’s, computed on its own 50-public/50-private partition, and is the only column that determines rank. They differ by 20-30%, which is why we do not quote local numbers as results.

23x end to end on AICrowd’s graded numbers (6.62e-06 → 2.88e-07), and 2.2x against the challenge’s own Monte-Carlo reference at equal compute.

The negative result, stated plainly

We spent most of our effort on the analytic branch and it was the wrong branch. Worse, we believed it was winning. Our notes recorded that the analytic estimator “beats Monte-Carlo sampling per-FLOP; sampling needs ~10k samples to match.” That claim came from misreading the starter kit’s compare_against_monte_carlo table, whose MSE column reports the estimator’s error against an n-sample MC reference, not the sampler’s error. It was never checked against the official scorer. When we finally ran a plain MC estimator through whest run, it beat our analytic estimator 4.4x.

A related trap: a local diagnostic comparing a cheap MC draw against a high-n MC “reference” is invalid if both use the starter kit’s default seed=0, because standard_normal((n, width)) fills row-major and the small draw is an exact prefix of the large one. That measures a subsample against its own superset. Both issues are filed separately as starter-kit feedback.

The methodological point: every measurement that misled us came from a local harness; every measurement that corrected us came from whest run against baked ground truth. When an estimator and its reference share an error source, the comparison is not evidence.

What we would do with more time

Sample count is exhausted and closure refinement is a dead end at this width, so the only remaining axis is variance per FLOP. Moment matching handled the first two moments; the natural successors are a Sobol/QMC sequence (uncertain at width 256, where QMC’s advantage degrades) and control variates built on the analytic estimate — which is cheap, correlated with the target, and currently unused. We would also expect a hybrid to be more robust than either pure branch on the private re-evaluation, since its bias and variance components fail in different ways.

LLM disclosure: the estimator code, the experiments, and this writeup were produced by Claude (Anthropic) working interactively with the participant. The participant is responsible for the accuracy and completeness of the final text. All reported numbers were generated by the official whest run / AICrowd grader, not by the model.

1 Like