[Phase 2 write-up] A bias/variance error budget for factored K=3 cumulant propagation: what the slices can tolerate, measured (submission #337475)
TL;DR. We re-derived and reimplemented the factored K=3 SIMPLE algorithm flat (≈200 lines of NumPy, machine-precision-identical to mlp_cumulant_propagation), then measured how much error, of what kind, injected where, the final-layer score can tolerate. Three quantitative results: (1) a σ² sensitivity law — iid perturbation of the third-cumulant slice read-outs of magnitude σ adds ≈ 0.9σ² (D21) / 0.6σ² (D3) to final-layer MSE, which puts a ~6%-relative-precision floor on any randomized slice estimator and closes the sketching family on cost grounds; (2) an eval-layer attenuation profile — the same noise injected only at early layers is tolerated at ~36× the variance of late layers; (3) a bias/variance asymmetry — low-rank truncation of the same read-outs, gated to the same early layers, gains only ~3.5×, because truncation error is bias that propagates through the state at gain ≈ 1 while zero-mean noise attenuates. We believe (3) is the mechanism behind the rank-“cliffs” reported by 504aldo and in the V37 sweeps: every truncation scheme is spending the (small) bias budget, while the (large) early-layer variance budget goes unused by every estimator family tried so far. We end with two directions this budget points at, including a cost argument that exact κ4 (2,1,1) read-outs can ride the existing K3 transport legs for ~one extra Hadamard-matmul per source-layer, rather than the ≥0.95×B of separate κ4 legs.
Graded submission: #337475 (score 2.966e-9, final-layer MSE 1.986e-8, utilization 0.1494, 50/50 scored). The submission itself is TristanMiano’s ALG058-V001 with koushik_rudra’s birth-column cut, both MIT, attribution chain in the file header (V29 © 504aldo, e30_a © LordProcciuto); our contribution is the measurements below, made with an independent reimplementation, not the estimator file.
1. An independent, exact, flat reimplementation (the measurement instrument)
To measure anything we first rebuilt the factored K=3 SIMPLE algorithm (arXiv:2605.05179, ARC’s mlp_cumulant_propagation) as ~200 lines of flat NumPy with no tensor-algebra framework: explicit ReLU Wick coefficients E[∂^k ReLU(Z)^p] for all needed (k,p) including k<p closed forms, the 15-term SIMPLE diagram list and pK→K conversion tables extracted programmatically from ARC’s own combinatorics, the factored (1,1,1) machinery with repeated-slice adjustments, and the radial κ4 scalar. Validated against the reference at n∈{16,32,48}, depth up to 16: max deviation ≲1e-13 relative. A “cohort” reorganization (every factor block = one evolving propagator P times a fixed birth matrix; two groups per source) is also exact to 1e-14 and is what all experiments below patch. At n=1024 our f32 flopscope port of the exact algorithm bills 1.40e12 FLOPs (63.7% of B) with no Strassen, after two small algebraic identities: the third birth block satisfies X3 = X1·diag(w21_birth) (so A3 = A1·w21_b is free), which also makes two of the three (2,1)-slice products equal, (A1∘B)A3ᵀ = (B∘A3)A1ᵀ — 9 → 7 dense units per source-layer. (V16b’s 4-unit decomposition subsumes this; we found it independently and note it here because the equality is not stated elsewhere.)
2. The σ² sensitivity law (variance budget)
Patch: add iid N(0,σ²) noise to the D3/D21 read-outs of all sources of age ≥2, every layer. n=256, depth 16, measured against the unperturbed estimate:
| σ on D21 | added final-layer MSE |
|---|---|
| 1e-5 | 9.1e-11 |
| 1e-4 | 9.1e-9 |
| 3e-4 | 8.2e-8 |
| 1e-3 | 9.1e-7 |
| 3e-3 | 8.2e-6 |
i.e. ΔMSE ≈ 0.91·σ² across four decades (D3: ≈0.60·σ²). With |D21| entries ≈5e-3 at n=256 (≈1.25e-3 at n=1024, entries scale ~1/n), holding ΔMSE below ~10% of the 2e-8 K3 floor requires ~6% relative precision per entry. A Gaussian sketch of a product XYᵀ has per-entry relative error ~√(n/k); 6% needs k ≈ 280n probes — more than the exact product. Any unbiased randomized estimator of the slices must beat √(n/k) variance scaling to be useful; none of the standard ones do. This closes sketching/subsampling/Hutchinson-type families quantitatively, complementing 504aldo’s empirical dead-end table with the law they obey.
3. The eval-layer attenuation profile (where the variance budget is slack)
Same noise, σ=1e-3, injected only while the current layer lies in a window:
| eval-layer window | added MSE | per-layer sensitivity |
|---|---|---|
| 0–5 | 4.8e-8 | ×1.0 |
| 6–9 | 2.9e-7 | ×9 |
| 10–12 | 5.7e-7 | ×24 |
| 13–14 | 4.7e-7 | ×30 |
| 15 | 2.9e-7 | ×36 |
Early-layer read-outs tolerate ≈36× the error variance of late ones (≈6× the amplitude). To our knowledge every rank schedule tried publicly is age-based; the eval-layer axis is a different, unmined coordinate.
4. The bias/variance asymmetry (why truncation hits cliffs)
The obvious exploit — run the early-layer read-outs at low rank — buys far less than ×36. Patch: project each old source’s read-out legs onto the top-r left singular subspace of its propagator for that layer’s read-out only (stored state stays exact), gated by eval layer:
| rank r | layers ≤7 only | all layers | gain |
|---|---|---|---|
| 32 | 3.8e-6 / 3.0e-6 | 1.3e-5 / 7.9e-6 | ~3× |
| 64 | 8.1e-7 / 1.1e-6 | 2.9e-6 / 1.5e-6 | ~2–3.5× |
| 96 | 5.0e-7 / 2.4e-7 | 8.1e-7 / 4.5e-7 | ~1.7× |
(two seeds, n=256.) Gating to the 36×-slack window helps only ~3.5×. The resolution of this apparent contradiction: truncation error is a bias — deterministic, correlated across layers, and it corrupts the covariance/κ3 state, which propagates forward at gain ≈1 instead of attenuating. The noise experiment measures the variance budget; truncation spends the much smaller bias budget. We verified the complementary fact directly at n=1024: distance-to-exact from stored-state compression adds to MSE essentially exactly (no cancellation), across (age, rank) configurations.
Consequences. (a) The “one notch lower is a cliff” rank laws are not a spectral accident; they are the bias budget binding. (b) The large early-layer variance budget is spendable only by estimators whose read-out error is zero-mean given the state — truncation, closures, and learned correctors all fail this; cheap sketches satisfy it but violate the §2 law. An estimator family with unbiased slice read-outs at o(n³) per source-layer and per-entry variance ≪ n/k would break the frontier; we conjecture bias-cancelling paired truncations (alternating-sign residual projections across layers, so state corruption telescopes) are the most promising concrete candidate and have not been tried.
5. Two measured dead ends (with mechanisms)
Diagonal-closure folding. Fold all sources older than A into aggregate slice state (d, u) propagated by the exact pair-collapse recursion u’ = W₂ᵀ(u_act + diag(d_act))W, d’ = 3·rowsum(W₂ᵀu₀ ∘ Wᵀ) − 2·W₃ᵀd_act, where W₂=W∘W, W₃=W₂∘W. This keeps every repeated-index term exactly and costs 2 matmuls/layer total; it still fails (fold at age 1: ΔMSE 3.0e-5 at n=256) because the dropped all-distinct triple contraction is the dominant information. Consistent with, and a sharper version of, the “memoryless closure” dead end: even keeping all pair-collapse structure through the true squared weights is insufficient.
Shared-basis aggregation with a common core. Aggregating all old sources into one rank-r basis with a pooled r³ core trades Σ_b n²r for nr³+r⁴ and only wins when (#sources)·n > r², i.e. below the rank cliff. Closed by arithmetic before implementation.
6. A cost argument for κ4 riding the K3 transport
The augmented-K3 accuracy step (κ4 beyond the memoryless λ·C_off regeneration) is worth ~1.5–1.9× in MSE (reference augmented K3; the 1.10–1.36e-8 leaderboard cluster). The known-failed costing gives κ4 its own transported legs (≥0.95×B). But the transport is the shared infrastructure: κ4 sources are born at the same ReLUs, from the same C0/Wick data, and transport by the same P as the K3 sources. The κ4 (2,1,1) read-out of a transported source has the form Σ_m (PX)ₐₘ(PY)ₐₘ(PZ)ᵦₘ — one more Hadamard-matmul on legs the K3 machinery already forms (A, B=P, Z). Marginal cost ≈ +1 dense unit per source-layer (~+25% on a 4-unit slot), not a parallel tensor chain; and because the κ4 channel can stay memoryless layer-to-layer, the §3 profile licenses computing these extra read-outs only at layers ≥12, cutting the addition to ~+8%. We have not implemented this to the grader; we publish the costing as the cheapest known-unexplored route to the leaders’ MSE tier.
7. Artifacts and attribution
The flat reimplementation, cohort harness, and all experiment scripts (noise law, attenuation windows, read-out projections, closure recursion, fold/truncation studies) are available on request / to be attached. Everything numerical was produced with LLM assistance under human direction. We built on and verify against: ARC’s mlp_cumulant_propagation (MIT); the V29 lineage — 504aldo (MIT), e30_a — LordProcciuto (MIT), ALG058 — TristanMiano (MIT), birth-column cut — koushik_rudra (MIT); Susupro’s K=4 measurements; and the prior Phase 2 write-ups, whose dead-end tables our budgets partially explain.