Submission ID: 336955 (Phase 2). The PDF of this write-up, submitted for the algorithmic-contribution prize, is attached: WRITEUP.pdf (286 KB). Code, graded folders and raw results: GitHub - yeongjunyoo/arc-whitebox-2026-phase2: ARC White-Box Estimation Challenge 2026, Phase 2 (submissions 336955 and 337488): a learned corrector on factored cumulant propagation, with the write-up, code and raw results · GitHub. The full text follows in this post and the next three.
Phase 2 technical write-up for submission 336955 (participant yeongjunyoo; author Yeongjun Yoo, 유영준).
Summary
Submission 336955 estimates the expected final-layer activations of the 1024 x 16 He-initialised ReLU networks without sampling. The grader scored it at 2.2984e-09 under flopscope 0.12.3: final-layer MSE 1.6616e-08 at 0.1383 of the FLOP budget. Our better designated submission, 337488 (the other is 335519, an earlier file kept for its wall-time margin), is this file plus later changes and scored 2.1878e-09 (final-layer MSE 1.4869e-08 at 0.1471): compression rank 256 with one more restriction (section 7.3), and il.miscusi’s scale-mode extraction in place of the memoryless law, with the corrector retrained (section 7.6). The repository’s submissions/shots.csv holds the grader’s values for every submission. Every ablation in this write-up measures 336955 unless a table says otherwise.
The engine is ARC’s factored K = 3 augmented cumulant propagation (Wu et al., arXiv:2605.05179; reference code mlp_cumulant_propagation, commit 93d091a, MIT), made affordable at this shape with structure team 504aldo published in forum post 18218: two dense legs per third-cumulant source, a memoryless one-scalar-per-layer law for the off-diagonal of the fourth-cumulant core, compressed old sources, and Strassen-Winograd written as flopscope operations. We re-implemented these from the paper, the reference code and the post’s prose. Section 2 says, part by part, whose idea each piece is.
We add three things:
- A learned corrector that reads the engine’s cross-neuron terms (section 4). After the engine has run, a small recurrent network proposes a correction to the mean at every layer from quantities the engine already computes. Among its inputs are eleven structure taps, per-neuron summaries of the engine’s cross-neuron terms (the off-diagonal third cumulant and pre-activation correlations on each neuron, the fourth-order folds of section 6, the leg energy lost to compression). Effect: x0.790 [0.776, 0.805] of the engine’s final-layer MSE on 48 corpus networks nothing of ours had touched, lower on all 48 (x0.798 on the 90 validation networks that selected it). Retrained without the structure taps it keeps x1.085 [1.075, 1.095] more error, on every seed, and with the neuron’s own-state inputs held at their mean x1.055 (section 4.5). The corrections are also carried forward through the engine’s mean-to-mean partial derivative, dp_t = D_t o (W_t^T dp_{t-1}) + u_t, but retraining finds no reliable effect of that carry (x1.005, with seeds on both sides of 1), and replacing its full readout derivative by the ReLU slope changes nothing (x1.0001). Lineage: it adapts mliston’s learned center correction (a GRU over the closure’s moment state with messages through W; forum 18170, Phase 1) to a sampling-free estimator. Susupro’s write-up (forum 18254, posted 2026-10-18) describes a corrector with the same linear carry; our graded submission 334200 (shot 11, 2026-10-05) already had it, earlier and independently, but since it is worth little here we do not count it as a contribution. What we claim is the inputs, the structure taps above all, the training and the measurements.
- Saturation-aware compute (section 5). Every large product runs only over the rows and columns of neurons above a layer-dependent threshold in a = m / sigma, with shapes fixed in setup and chosen per network, and dead birth neurons are gathered out of new sources. Restricting third-cumulant products to active neurons is also in koushik_rudra’s V34 (rows) and in Susupro’s extension to columns (forum 18254); we built ours independently (graded 2026-10-07). Our parts are the thresholds on rows, columns and the 3T^T band, the per-network choice among fixed shapes, and the birth rule. Effect, with the corrector off: adjusted x0.687 [0.679, 0.695] against the same engine with the restriction switched off (FLOPs x0.747, final MSE x0.919), lower on all 48 untouched networks. That it also lowers the MSE was not the aim. Section 5 measures what it is: the entries it drops are correct, so dropping them must offset error elsewhere in the engine. The ablations tie that error to the input-norm scale mode carried in the fourth-cumulant core rows: the scale-mode extraction of 337488 takes away most of the band’s gain (we did not trace the path).
- The star (3,1) fold (section 6). Several fourth-cumulant sources a layer’s nonlinearity creates live on the two legs of that layer’s third-cumulant source. For the young, uncompressed sources we fold the (3,1) slice of the “star” source into the product that already forms the third-cumulant matrix: no extra dense n^3 product, 0.44% more FLOPs. Kai_leal_gonzalez (forum 18246) measured that the (3,1) slice carries much of 504aldo’s estimator’s error (the exact slice lowers its final MSE by 18.1%); our fold carries one source’s exact share of that slice. Effect, with the corrector off: final MSE x0.982 [0.974, 0.990], adjusted x0.989 [0.981, 0.998], lower MSE on 35 of the 48 untouched networks.
Section 3 maps where the engine’s error is at width 1024: 78% of the last layer’s error comes in through the layer-14 mean; replacing the engine’s per-neuron variance, third and fourth cumulants by Monte Carlo truth at every layer leaves 8-10% of the final MSE, with nearly all of that value in layers 8-15. These diagnostics (on earlier engines, two to forty networks) led us to correct the mean at every layer and carry the corrections along the chain; section 4.5 later found that the gain comes from what the corrector reads, not from the carry.
How to read the numbers. Three kinds appear, and each table says which: (a) clean ablations of the submitted file on networks nothing was trained, tuned or selected on (section 7.1, and for 337488 the end of section 7.6); (b) acceptance reads during development, each on 48 public networks no candidate trained on but all candidates were judged on (section 7.5); (c) exploratory diagnostics on earlier engines (section 3) and on public training networks (section 5’s mechanism, section 7.6’s readable engine). Only (a) is confirmation; section 7.2 repeats (a) on networks that were used once for selection.
1. Setting and notation
Problem. Layers t = 0, …, 15 of width n = 1024: z_t = W_t^T h_{t-1} with h_{-1} = x ~ N(0, I), and h_t = ReLU(z_t). Weights are i.i.d. N(0, 2/n), no biases. The estimator returns E[h_15]. The score is the final-layer MSE against a 10^9-sample Monte Carlo reference, times max(0.1, C/B), where C is the FLOP count flopscope assigns analytically to the declared operations and B = 2^41. Above 0.1 B a 10% cut in cost is worth exactly a 10% cut in MSE.
Carried state (activation side, as in the reference): the mean mu_t = E[h_t], the covariance C_t, the third cumulant held as sources (below) and a fourth-cumulant core.
Pre-activation quantities of layer t. m = W_t^T mu_{t-1}, S = W_t^T C_{t-1} W_t (exact, since z_t is linear in h_{t-1}), sigma^2 = diag(S), a = m / sigma, S_off = S with its diagonal set to 0, t3_i = kappa3(z_i, z_i, z_i), d4_i = kappa4(z_i, z_i, z_i, z_i).
Wick coefficients. For Z ~ N(m_i, sigma_i^2), w_k = E[ReLU^(k)(Z)]: w_0 = sigma phi(a) + m Phi(a), w_1 = Phi(a), w_2 = phi(a)/sigma, w_3 = -a phi(a)/sigma^2, w_4 = (a^2 - 1) phi(a)/sigma^3, w_5 = (3a - a^3) phi(a)/sigma^4 (code: w01 … w51). Note d w_k / d m = w_{k+1}.
Readout. mu_t = w_0 + t3 w_3 / 6 + d4 w_4 / 24 per neuron.
Third-cumulant sources and legs. Each layer’s nonlinearity creates one source. A source is a sum over its birth neurons (“centers”) j of symmetrized products of three columns (“legs”) H_j, E_j, G_j. At birth G = I, H = diag(w_1) S_off and E = diag(x) + 3 Y^T, where x and Y are the new third cumulant’s repeated-index slices with the parts already carried subtracted (the reference’s construction). Between layers every leg is multiplied by diag(w_1) of its layer and then by W^T into the next layer; we call the W^T step the push.
T and 3T^T. T_ab = kappa3(z_a, z_a, z_b) for a != b. The engine forms 3T^T from all sources at once in one stacked product of “A rows” (Hadamard products of legs with coefficients) against the legs; the code calls it TT. Its diagonal is 3 t3. The next covariance reads T through the Wick (2,1) term (1/2)(w_2,a w_1,b T_ab + w_1,a w_2,b T_ba); the new source’s legs and the fourth-order rows read it too.
Fourth-cumulant core. The augmented term set carries the activation fourth cumulant through an n x n core Gamma = diag(gamma) + off-diagonal (the reference’s representation; section 2.2 says how we fill it).
2. The engine, and whose idea each part is
2.1 One layer, in the order the code runs it
The loop is _AugEngine.propagate in the submitted estimator.py. For layer t:
- Linear step. m = W^T mu, and S = W^T (C W) as a symmetric product by blocks (
_covsym), on some layers riding on the push’s Strassen-Winograd product. Then sigma, a, and the restricted sizes of section 5. - Push. Every stored source’s dense legs go through W^T in one stacked Strassen-Winograd product (
_sw), restricted to the rows and columns of section 5; thin legs by plain products; compressed sources are rebuilt from their bases. - 3T^T. One stacked product forms 3T^T (diagonal 3 t3) from all live sources, with the star-fold rows of section 6.
- Per-neuron algebra. Wick coefficients from exact polynomial tables in a, sigma, phi(a), Phi(a); the readout with t3 and d4; the diagonal of the fourth-cumulant core.
- Matrix stage. One einsum over a small table of diagrams gives the next covariance C’, the new source’s legs and the rows R4 that feed the fourth-cumulant core’s diagonal; the diag(w_1) factor of the legs is applied here (the G fold of section 6 just before it).
- Birth. The new source in two-leg form (section 2.2).
- Compression of the source that has just turned six layers old (section 7.3).
After the last layer the corrector of section 4 adds its correction to the final mean.
2.2 Parts taken from published work
| Part | Source | What we did with it |
|---|---|---|
| Factored K = 3 augmented propagation: term set, Wick/Hermite coefficients, source-and-leg factorisation | ARC: arXiv:2605.05179, mlp_cumulant_propagation (commit 93d091a, MIT) |
Ported to flopscope operations; checked against the reference numerically (per-layer means within 2.6e-9 relative at n = 16, 48 and 96, float64). |
| Two dense legs per source: the third leg written through the other two times diagonals, plus a low-rank residual from a randomized range finder | 504aldo, forum 18218, section 2.1 (M = P d(s) + 3 A d(e) + Z L^T, rank 16) | Our version: E = G diag(xv) + H diag(e) + Z L_z^T, where xv is E’s birth diagonal, e_j = <E_off[j], H[j]> / <H[j], H[j]> per birth neuron j (least squares; the post does not say how its d(e) is computed), and Z L_z^T is the best rank-4 approximation of the residual E_off - H diag(e) inside the range of (residual) Omega. Omega is the next layer’s first 32 weight columns: an i.i.d. Gaussian test matrix that costs no random numbers at prediction time. It does not select directions. |
| Memoryless fourth-cumulant core: Gamma_off = lambda’_t C_off, one scalar per layer, diagonal exact | 504aldo, forum 18218, section 2.2 | Our table LAM_REFIT = (0.01799, 0.01539, 0.01345, 0.01320, 0.01194, 0.01089, 0.00880, 0.00787, 0.00747, 0.00755, 0.00731, 0.00691, 0.00821, 0.00669, 0.00687, 0.00436) starts, like theirs, from a regression: the per-layer least-squares coefficient of the exact core’s off-diagonal on C_off, taken along the exact reference chain at n = 256 (one seed). We scaled it by 0.25 for n = 1024 as the 1/n power counting predicts (a scale scan of the final MSE on mini 50-52 put the optimum at 0.236-0.248), then refitted five layer-group scales (layers 0-2, 3-5, 6-8, 9-11, 12-14) by linearized least squares on finite differences of the final MSE on mini 50-52 (changes +26%, +28%, +7%, +17%, +46%), worth x0.939 [0.929, 0.948] of the final MSE on mini 0-39 at the time. Mini 51 and 52 are also in the corrector’s pool. The diagonal gamma is computed exactly each layer from row sums of the (4) and (2,2) slices. |
| Compressing old sources by age | 504aldo, forum 18218, section 2.4 (one shared, age-gated basis) | Each source is compressed separately (section 7.3). |
| Strassen-Winograd as flopscope operations, batched across sources, depth limited by the wall-time caps | 504aldo, forum 18218, section 2.7; the Phase 1 winners | Our recursive layout and in-place deepest level (section 7.4). |
When they went in, the augmented terms with the memoryless law cut the final MSE to x0.545 [0.528, 0.563] of our port of the reference SIMPLE engine on 40 mini networks (0-39), 40 of 40 better, at the same cost (the two are measured together; section 7.1 separates the law).
3. Where the engine’s error is (exploratory)
These are diagnostics on earlier versions of the engine; the version is named each time. None of them is a score.
Depth. The engine’s per-layer MSE grows almost linearly with depth: from 6.9e-10 at layer 0 (the Monte Carlo floor of the reference) to 2.33e-8 at layer 15 on 40 public networks with the engine of 10-04, and from 6.7e-10 at layer 1 to 2.07e-8 at layer 15 on the 90 validation networks with the engine of shot 34. Replacing only the layer-14 mean by the truth and recomputing layer 15 removes 78% of the layer-15 MSE of 2.33e-8 (1.81e-8): that much arrives through the layer-14 mean. The other 20% (4.62e-9) comes through the layer-14 covariance and cumulants and layer 15’s own closure; the two parts correlate at 0.028 (same 40 networks). Linear regressions on 32 layer-15 features explain 3.1% of the layer-15 error in sample. A correction that sees only the last layer cannot reach most of the error. 504aldo report the same from their side (18218, dead-end row 10, “Learned / online output corrections”: final-layer features correlate below 0.09 with the error).
The same oracle on the submitted engine. On 30 of the 90 validation networks we set only layer t’s output mean to the truth (its central moments kept) and ran the rest of the submitted file unchanged. The engine’s final-layer MSE falls to x0.991, x0.873, x0.504 and x0.192 of its value for t = 4, 8, 12 and 14 (30 of 30 networks better at t = 12 and 14). With the shipped corrector also left unchanged the final MSE is x0.993, x0.922, x0.679 and x0.431: trained on the engine’s usual errors, the corrector then corrects errors that are no longer there. The corrector itself leaves x0.75 of the mean MSE at layers 4 and 8, x0.77 at layer 12 and x0.80 at layers 14 and 15, so the information it has, not where it is applied, bounds what a mean correction can do. Putting the corrector’s own layer-t correction into the engine in place of the truth, and restarting its chain after t, leaves the final MSE where it was: x1.0003 [0.9999, 1.0008] at t = 14 and x0.9997 [0.9982, 1.0011] at t = 12, against the submitted file on the same 30 networks. The linear chain already carries that correction to the last layer, and nothing measurable comes through the variance path; without the restart the correction is counted twice (x1.203).
Channels. On two training networks (full 96 and 97) we measured every neuron’s first four cumulants at every layer from 2^25 Monte Carlo samples, replaced the engine’s own values by these at every layer, and carried the mean to the end (engine of 10-05, final-layer MSE 2.05e-8):
| True values injected at every layer | full 96 | full 97 |
|---|---|---|
| variance only | x0.708 | x0.577 |
| third cumulant only | x0.728 | x0.683 |
| fourth cumulant only | x0.717 | x0.731 |
| variance and third | x0.367 | x0.346 |
| variance, third and fourth | x0.099 | x0.082 |
Injecting at layers 1-7 only does nothing (x0.96 to x1.06); injecting at layers 8-15 only gives x0.11-0.12 for all three channels. Two networks are few, and the per-neuron Monte Carlo error of the third and fourth cumulants at 2^25 samples is of the same order as the engine’s own error on those neurons, so single-neuron comparisons are noise-limited; the injections average that noise over many neurons, but the noise they inject also adds error, so the 8-10% left is an upper bound on what the three channels leave (the MSE itself is against the dataset’s 10^9-sample means). The two diagnostics fit together: what reaches layer 15 through the layer-14 mean starts as closure error of the variance and cumulants in layers 8-14, carried forward through each layer’s readout. Nurlanggs (forum 18252) report the same picture from oracle injections into 504aldo’s chain (variance at layers 8-15: x0.687 and x0.647).
Neuron states. By the last layer’s a (engine and corrector of shot 34, the 90 validation networks):
| Last-layer state | Share of neurons | Share of the score |
|---|---|---|
| dead, a < -2 | 28% | 0.4% |
| band, abs(a) < 2 | 44% | 49% |
| on, a > 2 | 28% | 50% |
Width. Small-width runs of the reference code, depth 16, three seeds (so intervals are wide and several include 1), Monte Carlo truth with antithetic pairs (6.7e7 samples at n = 128, 1.34e8 at n = 256, 3.4e7 at n = 512; final-layer floors of at most 1.7e-9, 7.8e-10 and 1.9e-9, subtracted). Final-layer MSE as a ratio to factored K = 3 augmented, geometric mean and 95% t interval:
| Variant | n = 128 | n = 256 | n = 512 |
|---|---|---|---|
| factored K = 4 SIMPLE (reference) | 0.515 [0.264, 1.005] | 0.244 [0.157, 0.381] | not run (memory) |
| same, without the non-hypertree K = 3 terms that factored K = 3 drops | 2.10 | 0.838 [0.593, 1.183] | |
| those non-hypertree K = 3 terms with the fourth cumulant reduced to its trace part | 0.800 | ||
| fourth-cumulant sources up to age 1 read from shared legs (the factored K = 3 augmented terms plus the path, star, d4, d31, d22 and 3+2 kinds, without the d211 kind and the cross parts; not the shipped fold, which was not run at these widths) | 0.846 [0.651, 1.101] | 0.853 [0.795, 0.915] |
A separate closure that projects the full fourth cumulant onto the top-8 eigenspace of the activation covariance kept x0.715 [0.609, 0.839] of the error of its own trace-only control at n = 256 and x0.960 at n = 512. What these runs support, with their wide intervals: in the reference code the K = 4 gain over augmented K = 3 was larger at n = 256 than at n = 128; at n = 256 it appeared only when the fourth cumulant came together with the non-hypertree third-order terms; the pieces our representation can afford did not grow from 256 to 512; and the subspace closure lost most of its value from 256 to 512. Separately, Susupro (forum 18254) ran ARC’s exact unaugmented K = 4 at n = 1024 and report a raw final-layer MSE 11.8 times below augmented K = 3, a different algorithm from anything in our submission. Artyom_Mazur (forum 18249) report a closure that pays only together with a diagram the reference truncates.
4. Contribution 1: a learned corrector that reads the engine’s cross-neuron terms
4.1 Definition
After the engine has run, the corrector computes a correction dp_t for layers t = 1, …, 15 (dp_0 = 0) and returns mu_15 + dp_15:
dp_t = D_t o (W_t^T dp_{t-1}) + u_t,
D_t = w_1 + w_4 t3 / 6 + w_5 d4 / 24 = d mu_t / d m_t at fixed sigma, t3, d4,
u_t = 1e-4 sigma_t o (w_head . H_t + b_head), H_t = GRUCell(x_t, W_t^T H_{t-1} / sqrt 2), H_0 = 0,
with o the elementwise product. D_t o W_t^T is the partial Jacobian of layer t’s mean with respect to layer t-1’s mean, holding layer t’s variance and cumulants fixed. So dp_15 = sum_t (D_15 W_15^T … D_{t+1} W_{t+1}^T) u_t carries each local correction u_t to the last layer along the mean-to-mean path of the engine. The paths through the later covariances and cumulants are not included: a mean correction also moves w_1 and so the next covariance and sigma, and the chain leaves that path to the network, which sees it only through the (W o W)^T abs(dp) message. The chain changes only the mean. Section 4.5 finds no reliable effect of this carry once the network also passes its own state through W.
The input x_t of each neuron (code _wmsg, extra taps switch _QX):
| Group | Contents |
|---|---|
| own state (14) | a, sigma, phi(a), Phi(a), the t3 and d4 readout terms and their higher Edgeworth companions, higher ReLU cumulants of the neuron |
| structure taps (11) | the star slice’s diagonal and two shapes of the G fold (section 6); the off-diagonal energy and signed column sum of T on the neuron, also as the covariance update reads it; the leg energy lost to compression, carried forward; the neuron’s pre-activation correlation energy and signed sum |
| neighbourhood (16) | the previous layer’s ReLU cumulants and readout terms summed through the elementwise powers W^{o2}, …, W^{o6}; column norms of W |
| messages | W_t^T dp_{t-1} / sigma_t and (W_t o W_t)^T abs(dp_{t-1}) / sigma_t, and the depth |
The hidden size is 32 and the head is linear. The weights (wmsg.npz, 176 kB, with the input standardisation folded in) are loaded in setup(). At prediction time the corrector costs about 2.2 GFLOPs, 0.7% of the bill.
4.2 Lineage
| mliston (Phase 1, forum 18170 and its PDF) | Susupro (forum 18254, 2026-10-18) | This submission (shot 11, graded 2026-10-05) | |
|---|---|---|---|
| role | corrects the analytic center that a sample-based control variate then uses | adds a learned correction to a budget-pruned K = 3 chain | the whole correction; no samples |
| output per layer | the normalised mean correction of layer m, trained on mu_ref - a_base at layers 2-31 | the error the layer adds, carried forward linearly through the ReLU slopes and W | the local part u_t; the carried part D_t o (W_t^T dp_{t-1}) with the full readout derivative |
| earlier corrections | input messages Phi(alpha) o (W^T u_{m-1}) and an auxiliary output through W o W | a 32-dimensional state and the incoming correction, through W and W o W | input messages through W and W o W, and the carried term in the output |
| inputs | nine analytic channels, five sample-comparison channels | 35 chain statistics, 8 of the previous layer | engine state, structure taps, sums through W^{oK} |
| network | GRU, 96 hidden, 96-192-2 GELU readout | MLP, three layers of 128 GELU units | GRU, 32 hidden, linear head |
The structure (local correction plus a linear carry through the engine’s own mean derivative) was in our graded submission 334200 (shot 11) on 10-05; Susupro posted a corrector of the same structure on 10-18. We state that as chronology, not as a contribution: section 4.5 finds no reliable effect of the carry here, and its full readout derivative does no better than the ReLU slope Susupro use. What we claim as ours is the inputs, the structure taps above all (8.5% in section 4.5, against 5.5% for the neuron’s own state and 0.8% for the neighbourhood sums), the training and distillation, and the measurements of sections 4.4 and 4.5. The idea of learned networks that consume cumulant estimates as features is also in Jacob Hilton’s public LessWrong comment on the challenge announcement.
Before the corrector, shots 8 and 9 used two simpler learned parts, lc and law in the code; their flags are still on in the file, but the code skips both whenever the corrector runs. lc carried a corrected mean beside the engine, a’t = a_t + W_t^T (p’{t-1} - p_{t-1}) / sigma_t and p’_t = readout(a’_t) + net_t(features), with one small ReLU network per layer; law scaled the fourth-order law per neuron by 1 + c1 a + c2 a^2 with six fitted constants. The corrector replaced both in shot 11.
4.3 Training
Targets are the per-layer residuals r_t = truth_t - mu_t, from the dataset’s Monte Carlo means. The loss is
L = sum_t w_t MSE(r_t - dp_t) / MSE(r_t) / sum_t w_t, w_t = 1 for t < 15, w_15 = 5,
so every layer’s correction is trained, each layer’s error is normalised by the engine’s own, and the last layer counts five times. Adam (lr 2e-3, weight decay 1e-4), one network per step, up to 60 epochs with early stopping on mini 44-49 (patience 12). The shipped weights are a student of the same architecture trained on (1 - beta) r_t + beta dp_t^teacher with beta = 0.5, where dp^teacher is the mean of twelve correctors that differ only in their seed. Nine students (beta 0.5, 0.8 and 1.0, three seeds each) all passed the pre-set rule on the 90 validation networks, and the rule took the best. Appendix A lists every set of networks and its use.
4.4 Measured effect
| Comparison | Networks | Metric | Ratio | Kind |
|---|---|---|---|---|
| corrector on vs off (engine alone), submitted file | 48 untouched corpus networks | final MSE (floor-corrected) | x0.790 [0.776, 0.805], lower on 48 of 48 (adjusted x0.796) | clean |
| same | public one-shot selection set, 88 networks | final MSE | x0.802 [0.792, 0.812], lower on 88 of 88 (adjusted x0.808) | selection evidence (section 7.2) |
| same | 90 validation networks | final MSE | x0.798 (band x0.819, on x0.779; weakest cell a in [-1, 0) at x0.918) | picked the student |
lc, then law, when introduced (shot 8 against shot 7, shot 9 against shot 8) |
grader, 50 networks | final MSE | x0.879, then x0.974 | development |
| distilled student vs the single corrector of shot 33 | 90 validation networks | final MSE | x0.9817 [0.9762, 0.9874], 70 of 90 (the twelve-seed teacher x0.9785; its single members x0.988 to x1.004) | picked the student |
corrector replacing lc and law, when introduced (shot 11) |
full 0-47 acceptance read | final MSE | x0.9787 [0.9707, 0.9862], 35 of 48 | development |
lc and law were fitted once, on the engine of shots 8 and 9, and never refitted, so we do not compare them with the corrector on the submitted engine. The corrector was retrained seven times as the engine and its inputs changed (shots 23, 28 and 30-34). The shipped weights were trained on dumps of the engine of shots 30-32 (3T^T rows a > -1.5) and kept through the engine steps after it (push threshold 1.75, kept-subspace SVD, exact trims), so they never saw the submitted engine’s own features. Section 4.5 takes the corrector apart by retraining.
4.5 What the chain and the inputs are worth
Retrains with the same settings and pool except 30 epochs instead of 60 (every variant and its reference alike, to fit the time left), three seeds per variant (2, 3, 4; single correctors differ by about 2% from seed to seed), read on the 84 full networks of the validation set (the six mini networks there were used for early stopping), each against a retrained reference with everything on and the same seed. The plan and the variants (ladder.py) were fixed on 10-20 before any of these numbers existed. We report the mean over seeds of the paired ratio with a bootstrap over networks (seeds averaged inside it), and each seed’s own ratio, since the seed-to-seed spread is the larger uncertainty here:
| Change | Final MSE ratio against the reference retrain | Seeds 2, 3, 4 |
|---|---|---|
| no carried term in the output: dp_t = u_t (the dp messages stay inputs) | x1.0046 [1.0012, 1.0081] | 0.9994, 1.0124, 1.0021 |
| no carried term and no dp messages: only the hidden-state carry W^T H_{t-1} links the layers | x1.0052 [1.0017, 1.0086] | 0.9998, 1.0128, 1.0029 |
| carried term with D_t = w_1 only (ReLU slope, as in Susupro’s) | x1.0001 [1.0000, 1.0001] | 1.0001, 1.0000, 1.0001 |
| without the structure taps | x1.0847 [1.0749, 1.0947] | 1.0791, 1.0884, 1.0865 |
| the neuron’s own-state inputs (the 14 of section 4.1) held at their training mean | x1.0551 [1.0481, 1.0626] | 1.0488, 1.0587, 1.0580 |
| the neighbourhood inputs (the 16 sums through the powers of W) held at their training mean | x1.0082 [1.0046, 1.0120] | 0.9989, 1.0168, 1.0089 |
| without the messages | x0.9995 [0.9983, 1.0006] | 0.9974, 1.0019, 0.9991 |
| without the hidden-state carry (H_{t-1} := 0) | x1.0040 [0.9988, 1.0093] | 0.9987, 1.0062, 1.0072 |
The references keep x0.810 of the engine’s final MSE on these 84 networks (the shipped distilled student x0.798 on all 90, section 4.4).
Of the parts tested, two input groups matter. Retrained without the structure taps the corrector keeps 8.5% more of the error on every seed, and is better than its reference on only 2 to 7 of the 84 networks; with the neuron’s own state held at its mean it keeps 5.5% more, also on every seed (5 to 8 networks better). The neighbourhood sums add little (x1.008, one seed of three below 1). Each group was held out alone, so the others may cover for it, and the three effects need not add up to the corrector’s x0.81. We find no reliable effect of the linear carry: without it the mean is 0.5% worse, but one seed of three is better without it, with or without the dp messages. Removing the other link between layers, the network’s own state passed through W, costs as little (x1.004); we did not remove both. The full readout derivative D_t makes no measurable difference against D_t = w_1, and the messages none. So the claim of section 4.2 shrinks to the inputs: what lowers the error is what the network reads, not how its corrections travel between layers.
4.6 What we do not know
We cannot say exactly what the corrector has learned. The combination results (it replaces law and does not stack with it, x1.0051 when both are on) suggest it absorbs at least the systematic part the fourth-order law was fitting. That reading is a hypothesis. Section 4.5 shows which of its inputs it needs, the structure taps, not what it computes from them. We did not test feeding the corrections back into the engine’s covariance and cumulants.