Shipped covariance_propagation example trips the Phase 2 residual cap locally (0.16.0) but grades fine - what happens when the evaluator upgrades?

As the “Covariance propagation baseline scores 3.93e-7/3.93e-6” thread notes, several of us are on the board with the plain covariance propagation baseline, and it grades cleanly today. Running the same example locally under the kit’s own pins tells a different story, and the gap seems worth flagging before the evaluator upgrade lands.

Local result. examples/03_covariance_propagation.py, unchanged, on the Phase 2 mini split (hf://aicrowd/arc-whestbench-public-2026@v2-phase2, 100 MLPs, --runner local, whestbench 0.16.0 / flopscope 0.12.0): 2 of 100 MLPs fail on RESIDUAL (R_m = 0.689 s and 0.623 s against the 0.4 s cap). Each failure takes the zero-prediction fallback at MSE ~0.7, so the local adjusted score comes out 1.32e-02 instead of ~4e-07 - four orders of magnitude from two MLPs.

Mechanism. The example pulls flopscope arrays back into Python floats inside the layer loop: float(fnp.max(cov_diag)) in step 2 and float(fnp.exp(log_scale)) in step 8, every layer. Each float(...) is a synchronising round-trip to the flopscope backend, and the wait lands in the residual bucket, not in billed compute. At depth 16 that is 32+ round-trips per MLP. The absolute timings are machine-specific (this was a laptop, not the grader’s box), but the structure isn’t: every float() on a flopscope array is a round-trip whose latency is residual time.

A sync-free rewrite grades identically. Rescaling unconditionally instead of behind a Python if, accumulating log_scale as a flopscope scalar, and undoing the scaling once after the loop leaves the mathematics and the FLOP count alone (5.18e10 vs 5.17e10 per MLP) and gives 0/100 failures locally. Graded: 3.93e-7 (submission 328640), same as the baseline pack - consistent with the residual gate not binding on the evaluator today, which the kit CHANGELOG explains: the evaluator still runs whestbench@v0.15.0 + flopscope[server]==0.11.0, with PR #265 open to move it to the 0.16.0/0.12.0 pair.

Question for the organisers. When the evaluator moves to the pinned pair, does the 400 ms residual gate start binding - and if so, does it apply to future submissions only, or also to regrades of submissions that pass today? The overview limits post-start changes to “operational fixes applied consistently to all submissions”, so it would be good to know which side of that line the residual gate falls on. Entrants starting from the shipped example inherit this failure mode with no signal unless they run locally under the current pins.

Happy to open a PR against the starter kit with the sync-free variant of the example if that’s useful.

@konstantin_baltsat : Thanks, and please do send across a PR.
Also, the evaluators actually run on whestbench@v0.16.0 and flopscope[server]==flopscope[client]==0.12.0.

The changelog update in the starter kit fell through the cracks. Its nevertheless pinned properly in the starter kit, so if you are doing a uv sync in the repo, it should pick up the correct flopscope and whestbench versions.

Correction to my own question above, with fresh data: the residual gate IS enforced on the live evaluator today. The answer did not need to wait for the upgrade.

Two of my submissions, straddling the cliff:

  • Submission 328655 (~21.5k flopscope calls per MLP, moderate payloads): 100/100 MLPs completed, graded 1.24e-7.
    • Submission 328665 (same call count, ~15% larger per-call payloads): 5 of 50 public MLPs failed with RESIDUAL_WALL_TIME_EXHAUSTED, and five zero-fallbacks took the mean from ~9.4e-8 (the 45 completed MLPs) to 8.1e-2.
  • So the earlier framing was wrong in one respect: the baseline pack grades cleanly not because the gate is off, but because ~50 backend calls per MLP is far below it. A submission that makes tens of thousands of calls sits right at the 400 ms edge, and per-call payload size moves the residual clock too (serialization time lands in the residual bucket, not just the wait).

Practical summary for anyone building heavier estimators: residual budget on the evaluator is roughly (number of flopscope calls) x (per-call overhead + payload serialization). Fewer, larger operations are strictly better - batch your small pointwise ops. The local 0.16.0 runner’s gate is a good predictor of relative risk, but its absolute timings are machine-specific in both directions.

The original question to the organisers stands, but sharper: since the gate already binds, is the residual clock’s per-call overhead documented anywhere, and is it stable across grader load? The wall-time column on my two submissions varies 41-52 s for identical FLOPs, which suggests the residual measurement inherits some of that variance - and near the 400 ms line that variance decides pass/fail.