As the “Covariance propagation baseline scores 3.93e-7/3.93e-6” thread notes, several of us are on the board with the plain covariance propagation baseline, and it grades cleanly today. Running the same example locally under the kit’s own pins tells a different story, and the gap seems worth flagging before the evaluator upgrade lands.
Local result. examples/03_covariance_propagation.py, unchanged, on the Phase 2 mini split (hf://aicrowd/arc-whestbench-public-2026@v2-phase2, 100 MLPs, --runner local, whestbench 0.16.0 / flopscope 0.12.0): 2 of 100 MLPs fail on RESIDUAL (R_m = 0.689 s and 0.623 s against the 0.4 s cap). Each failure takes the zero-prediction fallback at MSE ~0.7, so the local adjusted score comes out 1.32e-02 instead of ~4e-07 - four orders of magnitude from two MLPs.
Mechanism. The example pulls flopscope arrays back into Python floats inside the layer loop: float(fnp.max(cov_diag)) in step 2 and float(fnp.exp(log_scale)) in step 8, every layer. Each float(...) is a synchronising round-trip to the flopscope backend, and the wait lands in the residual bucket, not in billed compute. At depth 16 that is 32+ round-trips per MLP. The absolute timings are machine-specific (this was a laptop, not the grader’s box), but the structure isn’t: every float() on a flopscope array is a round-trip whose latency is residual time.
A sync-free rewrite grades identically. Rescaling unconditionally instead of behind a Python if, accumulating log_scale as a flopscope scalar, and undoing the scaling once after the loop leaves the mathematics and the FLOP count alone (5.18e10 vs 5.17e10 per MLP) and gives 0/100 failures locally. Graded: 3.93e-7 (submission 328640), same as the baseline pack - consistent with the residual gate not binding on the evaluator today, which the kit CHANGELOG explains: the evaluator still runs whestbench@v0.15.0 + flopscope[server]==0.11.0, with PR #265 open to move it to the 0.16.0/0.12.0 pair.
Question for the organisers. When the evaluator moves to the pinned pair, does the 400 ms residual gate start binding - and if so, does it apply to future submissions only, or also to regrades of submissions that pass today? The overview limits post-start changes to “operational fixes applied consistently to all submissions”, so it would be good to know which side of that line the residual gate falls on. Entrants starting from the shipped example inherit this failure mode with no signal unless they run locally under the current pins.
Happy to open a PR against the starter kit with the sync-free variant of the example if that’s useful.