Hi all,
Have appreciated reading the write-ups so far, so figured I’d share mine. Like some of you I used a control variate plus a bag of tricks that Claude and Codex found along the way. The control’s anchor is an adapted form of cumulant propagation (truncated cumulant/Hermite closure through layer 18, cheaper Gaussian-reference tail after that). At layer 22 the estimator compares the anchor against the sample mean of a covariance-matched antithetic cloud (6–8.6k samples per MLP, chosen from the budget), then subtracts 0.6× the propagated difference from the final-layer sample mean.
The core idea is a bias/variance tradeoff in where you place the control. Deeper placement can explain more of the final-layer sampling variance, but the anchor gets more biased with depth as the truncated higher-order structure compounds. The experiment that pushed things in this direction was a diagnostic that centered the control with an essentially exact layer mean, estimated offline with a huge sample budget (fine for a diagnostic, far too expensive in a scored run). With that center a layer-24 control removed roughly 45% of raw MSE, while the same control centered with the plain closure mean got worse as it moved deeper. That suggested the deeper error signal is real and centering accuracy is the bottleneck. The mitigation was a small GRU (~52k params, frozen into the submission as constants) trained offline on a few thousand random MLPs from the challenge distribution to predict the closure’s centering error. It sees the closure’s moment state — mean, variance, third/fourth cumulants, gate probabilities, which the anchor pass already tracks as it propagates (a capped third-order factorization plus per-neuron fourth cumulants) — plus sample-shape statistics from the batch that gets drawn anyway. The direct sample-mean gap was deliberately left out of the inputs, since a model that copies that gap would presumably just erase the residual the control needs. In an on/off probe on 20 MLPs the frozen model cut raw MSE by 17–19%, though those are rough numbers given the panel size. That was enough to make layer 22 with a 0.6 coefficient the best accuracy-cost point in the grid search. Official Phase 1 public score: 1.334e-7.
Also one disclosure: my 8.62e-8 best on the public LB is not valid and was not among my selections - it benefits from a FlopScope cost-accounting issue (fnp.tensordot with contracted rank >52 bills multiplies only, so the adds in each FMA are free, ≈2× effective sample budget). The AICrowd team/@mohanty are already aware of it.
Submission #327723 Writeup.pdf (129.2 KB)