Sharing my Phase 1 write-up for submission #327838.
My angle was deliberately the unfashionable one. Reading the front page literally — “given the weights of a neural network, can you predict its expected per-neuron activations more accurately than running it many times?” — QMC is running it many times, so I spent the campaign on the other thing: a fully sampling-free propagator, deterministic in the weights, with no random draws anywhere in the forward pass.
The bet was bitter-lesson-shaped. Rather than hand-design a closure, bake high-quality data and let a scalable architecture learn the optimal propagator from it. So the first real work was the datasets, which are public:
- keenanpepper/whestbench-relu-mlp-means-500k — 500k train + 2k low-noise validation MLPs, per-layer post-ReLU mean targets, no weights stored (every MLP regenerates exactly from its seed via a spec-portable recipe). Target noise ~1.2e-8 train, ~7.5e-10 validation, the latter matching the official eval targets.
- keenanpepper/whestbench-relu-mlp-moments-10k — weights plus full pairwise moments to fourth order at N=1e8. This is what made the oracle/teacher-forcing measurements possible, and it’s the one I’d point at if you want to do this kind of work.
The estimator carries a mean/second-moment state, an O(N²) third-cumulant plane state standing in for the O(N³) tensor (whose exact transport bills 1.4–3.4× the entire budget), and a recurrent per-neuron latent, all trained end-to-end by backprop through the 32-layer recursion. It scores 1.50e-5 raw / 1.95e-6 adjusted, which is 4.8× under its own analytic backbone. The approach works.
The part I think is worth more than the artifact is what happened when I ablated my own submission. The learned attention closure the whole architecture was designed around is numerically inert — deleting it changes the prediction in the fifth significant figure while it consumes 61% of billed cost. A rebuild with that subtree removed is bit-identical and lands under the 0.1 multiplier floor unconditionally, i.e. I left a free 1.50e-6 adjusted on the table by shipping compute the network had learned to ignore. Roughly ninety numbered experiments went into a mechanism that trained itself to zero, and I only found out by ablating the shipped weights for the write-up.
The write-up also prices the whole family with oracle states. Inject true means and true covariance at every layer and the recursion bottoms out at 8.1e-7 raw — an oracle state genuinely does beat sampling at equal arithmetic, but because both methods sit near the multiplier floor that 18× raw advantage is worth only ~2× on the scored metric. So the honest answer to the challenge question at width 256 / depth 32 is no, not yet: sampling still wins, the barrier is state fidelity rather than the concept, and the entire remaining research program is worth about a factor of two. Wu et al. beat sampling at sufficient width to depth 12; I lose at depth 32. Whether that crossover is a width effect, a depth effect, or a depth/width effect is the measurement I’d most like to see someone do.
Also included: five measured walls, the negative results that shaped the design, and a methodology section on the practices that paid for themselves (pre-registered kill rules with numbers; through-train before believing any negative — frozen substitution misled me six times out of six).
AI disclosure: I directed the research and made the calls; Claude Opus 5 and Fable 5 did a large share of the coding, analysis, and drafting throughout, and the manuscript text plus the ablation/attribution/lean-rebuild measurements in it were produced by Opus 5 from the shipped artifact. Per the rules I’ve credited it as a co-author.
Feedback and criticism very welcome, especially from anyone who has pushed a learned propagator further than this.
main.pdf (681.5 KB)