A variance/bias budget for deep white-box mean estimation
ARC White-Box Estimation Challenge 2026, Phase 1. Algorithmic-contribution track (forum thread 18041).
Summary. The shipped estimator (internal tag v8a) predicts the per-neuron
post-ReLU mean of a depth-32, width-256 He-init ReLU MLP under N(0,I) inputs. It is a
Monte-Carlo-prefix / analytic-suffix hybrid: randomized quasi-Monte-Carlo (RQMC) sampling
through the first 31 layers, moment shrinkage toward an analytic Gaussian-closure chain, a
single Gaussian-conditional final layer, sample-skew and sample-kurtosis Edgeworth
corrections on the scored row, and an offline-trained, statistic-conditioned ridge
corrector. On the official public mini split (100 MLPs) it scores final-layer
\mathrm{MSE}=4.213\times10^{-6} (audit adf72b6fc316); on the grader it scores
adjusted 4.47\times10^{-7} (audit f0d2674631f3, multiplier 0.1009). That is about
6.3x better than the best plain-Monte-Carlo operating point (2.8\times10^{-6}
adjusted) and about 19x better than the bundled covariance-propagation baseline
(8.4\times10^{-6} adjusted).
None of the individual ingredients is novel in isolation. What we think is worth reporting
is the path: a measurement methodology (a two-budget variance/bias decomposition, with a
small lemma showing why it under-states bias under RQMC), a pre-registered promote/kill
discipline, and a long list of negative results, each with a mechanism rather than a number.
The negative results did more to shape the final estimator than the positive ones. This
writeup is organised around them.
LLM assistance is disclosed at the end (competition rules 5.7 / 6).
1. Problem, scoring, and why depth-32 is the hard part
Each task supplies the weights of a ReLU MLP (width n=256, depth L=32, He
initialisation) whose input is x^0\sim N(0,I_n). The estimator must return the full
(L, n) array of per-neuron post-ReLU means E[x^l_i], but only the final layer is
scored. The per-MLP score is
$$\text{score} = \text{final_layer_mse}\times\max(0.1,; C/B),$$
averaged over the suite (lower is better). B=2.72\times10^{11} FLOPs is the per-MLP
budget; C is the FLOPs the estimator actually spends (plus residual wall-time). Ground
truth is a baked N=10^9 Monte-Carlo estimate whose own noise floor (\sim2\times10^{-10})
sits far below any estimator error, so accuracy is the only lever. The grader is CPU-only,
has no scipy, and caps at 60s per MLP.
Two structural facts set the whole design:
(i) The compute floor. At the achievable accuracies, spending more than 0.1B never
pays: the multiplier is clamped at 0.1 below C/B=0.1, and above it the product
\text{mse}\times(C/B) rises faster than the mse falls once bias is nonzero (Section 3).
Every shipped configuration therefore sits at the 0.1 floor, and the raw final-layer mse
is the sole objective.
(ii) Depth destroys input-anchored structure. The organisers’ companion paper documents
that closed-form cumulant propagation degrades with depth. We measured the concrete cause
that kills the obvious variance-reduction tricks: the correlation between the true layer-l
activation chain and any surrogate anchored to the input decays polynomially in depth. For
the Gaussian-optimal affine surrogate (the natural control variate), the per-neuron coupling
correlation is 0.86 at 1 layer, 0.55 at 4, 0.41 at 8, and 0.25 at 32 (offline probe,
5 MLPs, audit 3fd6d9ea0b87). A control variate needs \rho\gtrsim0.95 to help; at depth 32
there is nothing left to couple to. This single measurement pre-emptively killed an entire
class of methods (Section 4) and pushed us toward estimating the deep-layer moments directly
by sampling, then closing the last step analytically.
Reference points (all reproduced locally before use): a zeros prediction scores \approx0.91
mse; the bundled mean-propagation example \approx9.5\times10^{-4}; the bundled
covariance-propagation example 8.37\times10^{-5} mse, i.e. 8.4\times10^{-6} adjusted
(audit 46ee57c9c5b8, matching the published calibration and confirming the local channel is
decision-grade). Plain Monte-Carlo tops out at \approx2.8\times10^{-6} adjusted at any
sample count, because above the floor the mse and the C/B multiplier cancel.
2. The estimator (v8a)
The pipeline is one forward object; each stage below states its own contribution with the
audit reference that measured it. The full-100 mini split is the decision channel throughout.
Code: exp/corrector_v3/ship_e17_a34_util098/estimator.py (byte-identical to the v7 champion
plus two corrector terms). Config: k=1, util=0.098, shrink=0.9, meanlam=0.95,
skewcorr=1.0, kurtcorr=1.0, RQMC roberts.
2.1 RQMC prefix. The input matrix (N,256) is drawn as a random-shift Kronecker lattice
(Cranley-Patterson rotation) rather than iid Gaussian: u_i=\mathrm{frac}(i\,\alpha+\text{shift}),
z_i=\Phi^{-1}(u_i), with the roberts generating vector \alpha_j=\varphi_d^{-(j+1)}
(\varphi_d the root of x^{d+1}=x+1). The shift is the only randomness, seeded from
mlp.seed, so the estimate is unbiased (same estimand as iid MC) and deterministic (two
runs byte-identical). norm.ppf exists in the grader’s flopscope (83 FLOPs/entry, about
0.44\% of the per-sample forward cost). Against an identical-budget iid-MC control the
lattice cuts final-layer mse from 9.307\times10^{-6} to 6.653\times10^{-6}, a 1.40x
reduction (audits 033ec1acdde6 vs 1a8b01f01eba). This is far short of the 5-7x that
competitors reported for RQMC on raw MC, and Section 3 explains why: the covariance
shrinkage below already removes most of the variance RQMC targets, so the two levers are
partly redundant. Antithetic mirroring was refuted at the probe stage (the z/-z pairing
perturbs the empirical covariance that feeds the analytic suffix). Detail:
exp/rqmc/RESULTS-rqmc.md.
2.2 MC prefix and the exact layer-1 row. The lattice is pushed through layers 1..L-1
with the real weights; each layer’s post-ReLU sample mean is recorded. The first row is
replaced by its closed form \sigma_1/\sqrt{2\pi} (the input is exactly Gaussian, so the
first post-ReLU mean is exact and free of sampling noise).
2.3 Moment shrinkage toward an analytic chain. In parallel we propagate an analytic
Gaussian-closure chain (\mu_a,\Sigma_a) using the exact rectified-Gaussian marginal
formulae and the leading-order gain method for the off-diagonal covariance. The empirical
layer-(L-1) moments are shrunk toward it:
\mu_s=0.95\,\mu_{\text{emp}}+0.05\,\mu_a,
\Sigma_s=0.9\,\Sigma_{\text{emp}}+0.1\,\Sigma_a.
The shrink is deliberately light (10% covariance, 5% mean). A heavier analytic weight injects
the analytic chain’s own bias: at \text{meanlam}=0.6 the score degrades to
1.57\times10^{-5} (audit 8ca5dd315284). The optimum \text{meanlam}=0.9 was
-15\% over the un-shrunk hybrid (audit dda34f3bca5a); the later retune to 0.95 is
marginal.
2.4 Gaussian-conditional final layer. The shrunk moments are propagated through the last
layer with the exact rectified-Gaussian mean E[\mathrm{ReLU}(z)]=\mu\Phi(\alpha)+\sigma\phi(\alpha),
\alpha=\mu/\sigma. This is the “Gaussian-closure” step whose residual bias the next two
stages attack. Derivation: exp/cumulant/DERIVATIONS.md Section 1.
2.5 Sample-skew and sample-kurtosis Edgeworth corrections. The final row gets two
Edgeworth terms, both computed from the layer-L pre-activation samples the pipeline already
has:
$$\delta_{\text{skew}}=-\frac{\kappa_3,\mu}{6\sigma^3},\phi(\alpha),\qquad
\delta_{\text{kurt}}=\frac{\kappa_4}{24},\frac{\alpha^2-1}{\sigma^3},\phi(\alpha),$$
with \kappa_3=\overline{z_c^3} and \kappa_4=\overline{z_c^4}-3\,\overline{z_c^2}^2 measured
per neuron. The signs come out of the Gram-Charlier expansion (opposite parity for the two
terms, He_3\phi=-\phi''' vs He_4\phi=+\phi''''); the derivations are in
exp/cumulant/DERIVATIONS.md Section 3 and exp/retune/DERIVATIONS-kurt.md, and both were
validated on a single MLP against a 2\times10^6-sample brute-force truth before any
full-mini run: the skew term removes 77.7% of the final non-Gaussian gap, skew+kurt removes
98.0% (audit f2dae80942e6). The skew term alone was -8\% (audit f8a649cbd9f1). The
important subtlety, discussed in Section 4, is that the analytically transported third
cumulant is nearly useless here; measuring it from the samples is what makes the Edgeworth
correction pay.
2.6 Offline-trained ridge corrector. The organisers confirmed at the town hall that
offline Monte-Carlo pretraining on locally generated MLPs is legal and untracked (it happens
at setup, not at grade time). We exploit this with a per-neuron, statistic-conditioned
correction \text{pred}_j \leftarrow \text{pred}_j + \sigma_{\text{pre},j}\,(\theta\cdot f_j),
where \theta is a 14-term ridge trained offline on 200 local MLPs (seeds disjoint from
the suite; N_{\text{truth}}=10^7 each). The features f_j are the pre-activation ratio
\alpha and its powers up to \alpha^4, the empirical-vs-analytic mean gap and covariance
gap projected through the last weight matrix, a per-neuron variance ratio, and a per-MLP
“glayer” descriptor (the MLP-mean per-layer moment gap, which identifies high-cumulative-shrinkage
MLPs and scales the whole correction). All features are low-variance at predict time, so the
correction reshapes systematic error without re-injecting Monte-Carlo variance.
The one methodological point worth extracting here (Section 3): an early version that fit the
corrector on converged (variance-free) features removed 60% of the converged bias offline
yet regressed the deployed score to 9.08\times10^{-6}, because the strongest bias
drivers, the empirical-vs-analytic moment gaps, are themselves noisy at the deployed
N\approx5500. Refitting on the noisy predict-time features (ridge then automatically
down-weights the low-SNR ones) is what ships. The corrector is worth about +18\% over the
skew-only pipeline (48-MLP fit, audit 22e729465e73), rising to +22.7\% at 200 MLPs with
the glayer feature (audit adbbcce963b0), holdout retention 95%. Detail:
exp/corrector/RESULTS-corrector.md.
2.7 Ship trajectory. Eight submissions, each adding one lever, tracked on both channels:
| ship | added lever | local mini mse (audit) | grader adjusted |
|---|---|---|---|
| v1 | MC-prefix hybrid + shrink |
9.880\times10^{-6} (af2ba3706fb7) |
9.14\times10^{-7} (87bc325ba0f6) |
| v2 | mean-shrinkage \lambda=0.9 |
8.372\times10^{-6} (dda34f3bca5a) |
8.53\times10^{-7} (2bb593443b00) |
| v3 | RQMC roberts prefix |
6.653\times10^{-6} (1a8b01f01eba) |
6.85\times10^{-7} (e1bfcbb0d58f) |
| v4 | sample-skew Edgeworth |
6.119\times10^{-6} (f8a649cbd9f1) |
6.14\times10^{-7} (cc97151e6e0b) |
| v5 | gap corrector (48 MLPs) |
4.985\times10^{-6} (22e729465e73) |
5.49\times10^{-7} (36c376b0a2b9) |
| v6a | glayer + 200-MLP fit |
4.733\times10^{-6} (adbbcce963b0) |
4.99\times10^{-7} (be35dcf0cc6b) |
| v7 | kurtosis term + retune |
4.535\times10^{-6} (82bfb8455f04) |
4.78\times10^{-7} (491efecfc59d) |
| v8a | \alpha^3,\alpha^4 tail + util 0.098 |
4.213\times10^{-6} (adf72b6fc316) |
4.47\times10^{-7} (f0d2674631f3) |
The grader/local ratio drifted from \approx0.8 on the early ships to \approx1.05 on the
later ones, because the early local numbers ran at a marginally inflated utilisation (an
aux_cost budget under-count, since fixed, let N overshoot by \sim6\%); after the fix the
grader is a touch harsher than local on its different 50-MLP suite, and the ratio is stable.
Every ship’s grader value carries its own audit reference; all eight landed inside their
pre-registered acceptance bands.
3. The measurement methodology that drove every decision
3.1 Two-budget variance/bias decomposition. Because everything sits at the compute floor,
the question “should I spend the unused compute on more samples?” is the same as “how much of
the residual is sampling variance?”. We answer it by running one configuration at two
utilisations and fitting the error model m(N)=b^2 + cN^{-p}. On the bare RQMC hybrid the
2x-budget run gave only -7\% (audit e36c36cbcf73): a naive 1/N fit reads \sim86\%
bias. On the v6a champion the 2x run read m_1=4.733\times10^{-6}\to m_2=3.732\times10^{-6}
(audit 6b92e6d83559), i.e. \text{bias}^2=2.81\times10^{-6}, about 59% bias under the
1/N model. Either way, marginal samples are nearly worthless and the floor is optimal:
doubling N would leave the floor and incur the multiplier penalty, so the equivalent
2x-util adjusted score is strictly worse.
3.2 A small lemma: the 1/N split under-states bias under RQMC. The two-budget estimator
of bias is
$$\hat b^2(p) = m_1 - \frac{m_1-m_2}{1-q^{-p}},\qquad q=N_2/N_1>1.$$
Since 1-q^{-p} is increasing in p, the subtracted term is decreasing in p, so
\hat b^2(p) is increasing in p. RQMC beats iid, i.e. its variance decays faster than
1/N (p>1), therefore the iid assumption p=1 yields a lower bound on the true bias.
Numerically, on the v6a run: p=1 gives \text{bias}^2=2.81\times10^{-6} (59.4%), p=1.5
gives 3.24\times10^{-6} (68.4%), p=2 gives 3.43\times10^{-6} (72.6%) (audit
6b92e6d83559). Our own pre-registration had this backwards (it warned that the 1/N split
over-states bias); the algebra and the numeric check both show the opposite. The residual is
even more bias-dominated than the naive split suggests, which is why the entire second half of
the campaign chased bias, not variance.
3.3 Pre-registered promote/kill bars. Before any result existed, each experiment wave was
committed to a numeric bar in the campaign log (e.g. “E8 RQMC promotes iff full-mini mse
<8.4\times10^{-6}”, “E12 K=3 must explain \ge5\% of the residual on the decisive test”).
Bars were single-shot and could not be re-rolled. Where a bar was missed on a technicality
(e.g. v6a cleared “5% under v5” at 5.06% but nominally sat 0.05\% above the rounded
4.73\times10^{-6}), both readings were recorded and the substantive criterion governed, with
the ruling logged. This is deliberately paranoid: the failure mode of a solo agent optimising
a proxy is talking itself past its own bars.
3.4 Full-suite-only gating, after screens inverted. Early on, a 5-MLP screen for the exact
Wick covariance (E2) reported K\ge2 worse than K=1 and would have false-killed a real
+6.5\% lever; the full-100 channel showed the opposite (RESULTS-cumulant.md, gotcha 1).
From that point every kill and every promote was decided on the full 100-MLP split, never on a
screen. The one thing screens were still trusted for was the cheap decisive test before a
build (below), because those tests measure a structural quantity (a coupling correlation, a
fraction-of-residual), not a noisy score.
3.5 Decisive cheap tests before expensive builds. The most useful discipline was refusing
to build anything whose premise could be falsified offline for a few CPU-minutes. The coupled
surrogate CV (Section 4.1), the low-rank K=3 cumulant method (4.2), the eigen-ordered lattice
(4.6), and the learned quadrature weights (4.5) were each killed by a one-file numpy probe
before a single grader submission was spent on them. Four whole method families, zero slots.
4. Negative results
The judges asked for these explicitly, so here is the catalogue, grouped by the mechanism that
kills each one. Every entry is a hypothesis we genuinely expected to work.
4.1 Depth decouples input-anchored structure
Coupled affine surrogate control variate (E10, dead by decisive test). Hypothesis: a
cheap Gaussian-optimal affine surrogate chain, run on the same inputs, is a control variate for
the true chain. Test: measure the per-neuron coupling correlation at depth 32. Result:
\rho=0.25 (gate required 0.95), decaying 0.86/0.55/0.41/0.25 at layers 1/4/8/32 (audit
3fd6d9ea0b87). Mechanism: ReLU mixing decorrelates any input-anchored surrogate from the
true state polynomially in depth. No build.
Eigen / PCA-ordered lattice (L4, dead by decisive test). Hypothesis: draw z_1 directly
with the largest eigen-directions of W_1^\top W_1 mapped onto the best lattice dimensions.
Test: variance ratio on a 24-shift, 25-MLP probe. Result: 0.95-$1.04$x (gate $1.1$x), the
higher-power read landing marginally below 1 (audit in exp/wave_l2/RESULTS-L4.md).
Mechanism: the roberts lattice is deliberately dimension-balanced (\alpha_j ranges only
0.996\to0.369 across 256 dims, all O(1)), so there is no per-dimension quality gradient to
exploit by reordering, unlike a scrambled Sobol’ net; and 30 layers of mixing wash out any
layer-1 alignment anyway. The pre-registered K=8 point estimate ($1.245$x) cleared the bar
but was pure noise (its own CI included 1.0); we reported the underpowered screen honestly and
let the high-power read govern.
4.2 The skew is real but the analytic transport and the marginal are both dead ends
This is the E12 to E14 pivot, and the most instructive result of the campaign.
Diagonal third-cumulant transport (E3, killed). Propagating the diagonal third cumulant by
\kappa_3^{\text{pre}}=(W^{\odot3})^\top\kappa_3^{\text{post}} and Edgeworth-correcting the
marginal gave only -1.44\% at the final layer (bar 5\%; audit caae794dd6c3). The
transported standardized skew looked tiny (rms 0.006-0.033). We nearly concluded skew was
negligible.
It was not: the transport was lossy. The decisive K=3 probe (E12) brute-forced the true
final-layer skew and found it large: standardized g_1 rms 0.455, max 0.87 (audit
fdf77254e148). The diagonal transport had dropped the cross third-cumulants, which carry
essentially all of the skew: diagonal-only propagation loses about 98% of it. So the
premise “skew matters” was right; the way we were getting the skew was wrong.
But the marginal mean is skew-insensitive anyway. The same probe decomposed the analytic
mean residual: the propagation error in (\mu,\sigma) is 99\% of it, and final-layer
non-Gaussianity only 2\%; even an exact diagonal \kappa_3 Edgeworth term moves the
scored mean by 1.59\% of the residual (bar 5\%). Two structural reasons, both provable:
(a) E[\mathrm{ReLU}(z^L_i)] depends only on the marginal law of z^L_i, so cross cumulants
\kappa_3[i,i,j] cannot touch it by construction (derivation in
exp/k3lowrank/DERIVATIONS-k3.md Section 1). This kills the organiser-suggested low-rank /
sliced / CP third-cumulant menu for the final mean: everything beyond the diagonal encodes
cross-cumulants that matter only for deeper propagation. (b) the ReLU mean is a near-linear
functional of the low-order moments, so a third-order perturbation is intrinsically small.
The pivot (E14, promoted). If transport loses the skew and the marginal is skew-insensitive
to transported cumulants, but the true skew is large, then measure \kappa_3 directly from
the samples the pipeline already draws (SE \approx0.03) and apply the same Edgeworth term.
That is exactly stage 2.5, and it pays -8\%. The lesson: a correction can be simultaneously
“structurally tiny for the mean” (E12) and “worth 8% of the score” (E14), because the transport
error dwarfs the marginal-sensitivity argument. The kill and the promote are not contradictory;
they measure different things.
4.3 Bias-anchor traps
Layerwise mean re-anchoring (E11, dead, all configs). Hypothesis: nudge the sample cloud
mean toward the analytic covprop mean after every ReLU to damp accumulated sampling noise.
Result: 4-$9$x worse, monotone in anchor strength (7.26\times10^{-5} at \lambda=0.8
down to 3.12\times10^{-5} at \lambda=0.95; audit 1e0197ddca2c). Mechanism: the covprop
mean is biased, and injecting a biased anchor at every layer compounds that bias through all
remaining layers. Shrinking toward a biased chain is only safe as a light, terminal touch
(stage 2.3), never layer-by-layer. An unbiased anchor would rescue it; none exists.
Converged-feature corrector fit (E9a, dead). Covered in 2.6: fitting the corrector on
variance-free features removed 60% of converged bias offline but regressed the deployed score
to 9.08\times10^{-6}, because the dominant bias drivers are noisy at deploy time. Fit on the
noisy predict-time features, not the clean ones.
4.4 Robustness that is not
Median-of-means moments (E7, killed). Block-median of the per-neuron mean over 5 or 10
blocks scored 1.31\times10^{-5} and 1.23\times10^{-5}, both worse than the plain mean
(audits 8ef2f62d1490, 0e2ee84903b7). Mechanism: block-median inefficiency exceeds its
outlier protection, and the two-budget decomposition had already shown the worst-MLP outliers
are pure variance, not sample-level corruption, so there is nothing for a robust estimator to
protect against.
Antithetic RQMC (refuted at probe). The z/-z pairing is worse than the plain lattice at
both generating vectors, because it perturbs the empirical covariance estimate feeding the
analytic suffix.
4.5 Learned-weight and nonlinear over-reach
Offline-learned quadrature weights (L2, dead by decisive test). Hypothesis: replace the
1/N average with weights learned offline against stored truth, made unbiased by a per-MLP
Haar rotation of a fixed lattice. Result: max +1.7\% held-out gain (bar 8\%; the
ridge-optimal weights collapse to uniform; exp/wave_l2/RESULTS-L2.md). Mechanism, and this is
the elegant part: the very rotation that makes the learned weights legal (unbiased) also
Haar-scrambles the per-MLP directions, so no cross-MLP weight signal survives. “Do not
reweight” is optimal precisely because the randomization is well-designed.
Per-neuron nonlinear corrector heads (E17, underperform the polynomial). A tanh-MLP head
gave -2.95\% holdout (a from-scratch, non-zero-init variant blew up to +48\%, retention
0%); depth-1 GBM stumps gave -2.0\%; the plain ridge with \alpha^3,\alpha^4 added gave
-3.76\% (retention 98.5%), and \alpha^5,\alpha^6 add nothing (audits in
exp/corrector_v3/RESULTS-v3.md). Mechanism: the reachable residual is a low-order polynomial
in the pre-activation ratio \alpha, living in its tail (57\% of neurons have
|\alpha|>3, range to \pm6.5). A localized RBF basis reaches only -1.31\% and clipping
\alpha to \pm4 destroys the gain; the correction needs the full tail range that bounded /
saturating models cannot represent. The flexible net does not learn more than the polynomial,
it learns less, and overfits the Monte-Carlo-noisy residual on the way. The per-neuron detail
the task hypothesized (weight-column norms, per-neuron skew/kurtosis shapes) is dead
(\approx0\% signal beyond the polynomial).
4.6 Sub-dominant terms confirmed sub-dominant
Plug-in curvature (delta-method) term (L5iii). The leading delta-method bias
\phi\,v_e/(v_{\text{pre}}N) has RMS 4.6\times10^{-6} in output units, about 3 orders below
the residual RMS 2.15\times10^{-3}, correlation +0.019; ridge zeros its coefficient (audit
14bb85830743). RQMC already cut the sampling variance, so the residual is model (closure)
bias, not plug-in curvature.
Exact Wick off-diagonal inside the hybrid (E5 / L5i). The exact Price/Wick series for the
ReLU cross-covariance is a real -6.5\% improvement for the pure covariance-propagation
estimator (audit f5d68fde948c, versus the leading-order gain method, which we verified is
exactly the k=1 truncation of the series). But swapped into the hybrid’s analytic anchor it
buys +0.36\%: the hybrid leans on the empirical MC covariance (shrink \to1.0), so the
accuracy of the analytic anchor is nearly irrelevant there (exp/wave_l/RESULTS-L.md). A clean
example of a lever that matters in one regime and vanishes in another.
Global / cross-neuron corrector features (L1). 20 MLP-level spectral and weight-chain
descriptors added +0.16\% (bar 5\%; audit 92433090577d), because the reachable MLP-level
signal is already saturated by the glayer feature; the residual is per-neuron alpha-tail, not
global-spectral.
Fixed-shift derandomization (L3). A single pre-committed Cranley-Patterson shift gave
+4.43\% (bar 5\%), but the effect swings from -0.14\% to +8.91\% across shift seeds and
is catastrophic (-311\%) unshifted. Real but shift-dependent, killed on the pre-committed
shift with no bar-shopping.
5. Where the residual error lives
Taking stock of the 4.2\times10^{-6} final-layer mse:
It is bias-dominated. The two-budget decomposition (3.1, 3.2) puts \ge58\%, and under the
realistic RQMC decay (p>1) 68-73\%, into closure bias; the rest is sampling variance that
cannot be cut without leaving the compute floor and taking the multiplier penalty. The variance
side is a hard floor at b^2; the headroom is all on the bias side.
The bias-side corrector has saturated at MLP-level statistics. The learned corrector
reaches every predictable component that is a function of the observable per-neuron ratio
\alpha (a saturating \alpha^3,\alpha^4 tail-polynomial) and the per-MLP moment-gap
descriptor (glayer). Beyond that, three independent probes (global spectral features L1,
nonlinear heads E17, plug-in curvature L5iii) all return \approx0\%. What remains is the
Gaussian-closure bias of the analytic suffix itself, i.e. the error of approximating the deep
pre-activation law by its first two moments plus two Edgeworth terms. That error is not a
function of any cheap per-neuron statistic; it is a property of the propagation, and no
corrector applied at the final layer can reach it.
The open problem this motivates. The remaining bias lives in the moment chain, not the
final marginal. Cutting it needs a better deterministic propagation of the joint law
(closures for the covariance and higher cumulants that stay accurate at depth 32), not a
richer sampler or a richer terminal corrector. This is precisely the direction the organisers
named at the town hall (low-rank cumulant refinement as the most promising long-term path, with
the caveat that the naive Edgeworth series diverges and needs resummation) and is the axis we
have moved to for Phase 2: deterministic cumulant propagation with offline-learned per-layer
closure corrections, targeting the propagation error that stages 2.3-2.6 can only partially mask.
The honest position at Phase 1 is that the sampling-plus-closure hybrid is close to the ceiling
of what its architecture can reach; the next factor is a different kind of estimator.
6. Reproducibility
Determinism. The only randomness is fnp.random.default_rng(mlp.seed), used for the single
lattice shift. Two identical full-mini runs of the champion returned bit-identical final-layer
mse (4.212975700284\times10^{-6}); the grader supplies a fixed per-MLP seed and prize
submissions must reproduce under it.
Rerun a shipped number. Each ship folder is self-contained (estimator.py plus the
weights_rqmc_v2.npz corrector, loaded at setup, 0 FLOPs at grade time) and reproduces its
documented score with no environment variables set (grader emulation confirmed exact). The
decision channel is whest run --split mini on the official public 100-MLP split.
Retrain the corrector. exp/retune/build_refit_cache.py caches the shrink/meanlam-independent
RQMC forward moments for 200 offline MLPs (R=10 realizations, $\sim165$s, peak RSS $\sim0.6$GB);
exp/corrector_v3/fit_ship_v3.py refits the ridge and reproduces the shipped coefficients to
max-abs-diff 0.0. The offline generation is exp/corrector/gen_offline.py (memory-hardened:
niced workers, streaming batches, per-MLP incremental writes).
Audit trail. Every number in this writeup carries either an audit reference (an entry in the
append-only campaign audit) or a pointer to the per-wave ledger:
exp/RESULTS-cumulant.md (E2/E3), exp/rqmc/RESULTS-rqmc.md (RQMC), exp/k3lowrank/RESULTS-k3.md
(K=3 decisive test), exp/corrector/RESULTS-corrector.md (corrector), exp/retune/RESULTS-retune.md
(kurtosis and retune), exp/corrector_v3/RESULTS-v3.md (tail-polynomial and nonlinear heads),
exp/wave_l/RESULTS-L.md and exp/wave_l2/RESULTS-L2.md / RESULTS-L4.md (final negative
wave). Derivations are in exp/cumulant/DERIVATIONS.md, exp/k3lowrank/DERIVATIONS-k3.md, and
exp/retune/DERIVATIONS-kurt.md.
Compute. All numbers come from local CPU (no paid compute) and the official grader. Every
shipped configuration runs at the 0.1 FLOP floor; the FLOP accounting for the RQMC input path
(norm.ppf 83 FLOPs/entry, force-upcast to float64 and un-dodgeable) and the corrector
(\Delta C/B\approx1.2\times10^{-4}) is documented in the RQMC and corrector ledgers.
DISCLOSURE
This solution and this writeup were developed with LLM assistance (Claude), as permitted and
required to be disclosed under competition rules 5.7 / 6. The LLM proposed and implemented the
estimator code, ran the experiments, and drafted this document; a human directed the campaign,
set the pre-registered bars, and reviewed the results. All measurements are from the official
evaluation pipeline (local mini split and the AIcrowd grader); the final numbers and the
determinism checks were verified by rerun. No proprietary data or paid compute was used.
CHANGELOG
- 2026-07-21: Full technical draft written from the campaign ledgers (STATE / RESULTS / LOG and
the per-waveexp/*/RESULTS-*.mdandDERIVATIONS-*.md). Replaces the skeleton. Covers the
shipped v8a pipeline, the two-budget measurement methodology and the RQMC bias-underestimate
lemma, the full negative-result catalogue with mechanisms, the residual-error accounting, and
reproducibility. Headline improvement factors stated against the plain-MC ($6.3$x) and
covariance-propagation ($19$x) adjusted baselines, both traced to the calibration ledger.