Shipped covariance_propagation example trips the Phase 2 residual cap locally (0.16.0) but grades fine - what happens when the evaluator upgrades?

ARC White-Box Estimation Challenge 2026
1

As the “Covariance propagation baseline scores 3.93e-7/3.93e-6” thread notes, several of us are on the board with the plain covariance propagation baseline, and it grades cleanly today. Running the same example locally under the kit’s own pins tells a different story, and the gap seems worth flagging before the evaluator upgrade lands.

Local result. examples/03_covariance_propagation.py, unchanged, on the Phase 2 mini split (hf://aicrowd/arc-whestbench-public-2026@v2-phase2, 100 MLPs, --runner local, whestbench 0.16.0 / flopscope 0.12.0): 2 of 100 MLPs fail on RESIDUAL (R_m = 0.689 s and 0.623 s against the 0.4 s cap). Each failure takes the zero-prediction fallback at MSE ~0.7, so the local adjusted score comes out 1.32e-02 instead of ~4e-07 - four orders of magnitude from two MLPs.

Mechanism. The example pulls flopscope arrays back into Python floats inside the layer loop: float(fnp.max(cov_diag)) in step 2 and float(fnp.exp(log_scale)) in step 8, every layer. Each float(...) is a synchronising round-trip to the flopscope backend, and the wait lands in the residual bucket, not in billed compute. At depth 16 that is 32+ round-trips per MLP. The absolute timings are machine-specific (this was a laptop, not the grader’s box), but the structure isn’t: every float() on a flopscope array is a round-trip whose latency is residual time.

A sync-free rewrite grades identically. Rescaling unconditionally instead of behind a Python if, accumulating log_scale as a flopscope scalar, and undoing the scaling once after the loop leaves the mathematics and the FLOP count alone (5.18e10 vs 5.17e10 per MLP) and gives 0/100 failures locally. Graded: 3.93e-7 (submission 328640), same as the baseline pack - consistent with the residual gate not binding on the evaluator today, which the kit CHANGELOG explains: the evaluator still runs whestbench@v0.15.0 + flopscope[server]==0.11.0, with PR #265 open to move it to the 0.16.0/0.12.0 pair.

Question for the organisers. When the evaluator moves to the pinned pair, does the 400 ms residual gate start binding - and if so, does it apply to future submissions only, or also to regrades of submissions that pass today? The overview limits post-start changes to “operational fixes applied consistently to all submissions”, so it would be good to know which side of that line the residual gate falls on. Entrants starting from the shipped example inherit this failure mode with no signal unless they run locally under the current pins.

Happy to open a PR against the starter kit with the sync-free variant of the example if that’s useful.

2

@konstantin_baltsat : Thanks, and please do send across a PR.
Also, the evaluators actually run on whestbench@v0.16.0 and flopscope[server]==flopscope[client]==0.12.0.

The changelog update in the starter kit fell through the cracks. Its nevertheless pinned properly in the starter kit, so if you are doing a uv sync in the repo, it should pick up the correct flopscope and whestbench versions.

3

Correction to my own question above, with fresh data: the residual gate IS enforced on the live evaluator today. The answer did not need to wait for the upgrade.

Two of my submissions, straddling the cliff:

  • Submission 328655 (~21.5k flopscope calls per MLP, moderate payloads): 100/100 MLPs completed, graded 1.24e-7.
    • Submission 328665 (same call count, ~15% larger per-call payloads): 5 of 50 public MLPs failed with RESIDUAL_WALL_TIME_EXHAUSTED, and five zero-fallbacks took the mean from ~9.4e-8 (the 45 completed MLPs) to 8.1e-2.
  • So the earlier framing was wrong in one respect: the baseline pack grades cleanly not because the gate is off, but because ~50 backend calls per MLP is far below it. A submission that makes tens of thousands of calls sits right at the 400 ms edge, and per-call payload size moves the residual clock too (serialization time lands in the residual bucket, not just the wait).

Practical summary for anyone building heavier estimators: residual budget on the evaluator is roughly (number of flopscope calls) x (per-call overhead + payload serialization). Fewer, larger operations are strictly better - batch your small pointwise ops. The local 0.16.0 runner’s gate is a good predictor of relative risk, but its absolute timings are machine-specific in both directions.

The original question to the organisers stands, but sharper: since the gate already binds, is the residual clock’s per-call overhead documented anywhere, and is it stable across grader load? The wall-time column on my two submissions varies 41-52 s for identical FLOPs, which suggests the residual measurement inherits some of that variance - and near the 400 ms line that variance decides pass/fail.

4

Second correction, and this one matters more than the first: the residual clock is dominated by client-side synchronisation points, not by call count or payload size.

After instrumenting the client, the failure mechanism in my rank-scaled submission turned out to be graph flushes: every bool(), float(), or int() on a flopscope value (including implicit ones - an if on a comparison result, an assert on array data, allclose checks) blocks the client until the entire pending operation graph has executed on the backend. All of that waiting is charged to the 400 ms residual budget. Larger operands made each flush longer, which is why the failure correlated with rank - but the flushes themselves were the cost.

My estimator had 166 such sync points per MLP: 119 from a defensive invariant re-check (an assert on array data in an inner loop) and 47 from a float() on a value that was a Python constant upstream. Removing all of them - same mathematics, same FLOP count, ~21k backend calls unchanged - took the same configuration from 5/50 RESIDUAL failures to a clean 50/50 pass (submissions 328665 vs 328702).

So the practical rule is stricter than “batch your small ops”: never force a materialisation mid-computation. Keep every branch condition and every scalar coefficient in plain Python, let the whole per-MLP graph pipeline server-side, and only the returned prediction should ever cross back. Call count still adds per-call overhead, but at ~21k calls it fits comfortably once the flushes are gone - one flush over a deep pending graph costs more than thousands of async enqueues.

Instrumentation that finds these in your own estimator: wrap FlopscopeArray.__bool__ / __float__ / __int__ with a counter and run one MLP locally. If the count is not zero, each hit is a residual-budget stall on the grader.

5

@mohanty One more data point for this thread, and a question about a submission-level failure mode.

Two of my submissions today failed with the generic “Error : Error while scoring your submission” - not a per-MLP failure (no per-MLP rows were produced at all). The same code passes the full public mini split locally under both flopscope 0.12.0 and 0.11.0 in-process. The distinctive features of the failing submissions relative to my graded ones: batched 3-D einsum calls (einsum("bij,bjk->bik", ...) with a leading batch dim up to 343) and larger single operands (stacked arrays up to ~1.1 GB) crossing to the flopscope backend.

Could you say what the evaluator-side error actually was for those two (worker OOM, a request/message size limit on the client-server wire, an unsupported op route on flopscope[server]==0.11.0, or a plain crash)? Happy to share submission IDs privately if that helps. Knowing which constraint we hit would save every team that tries batched formulations from burning submissions on it - and would tell us whether the constraint is intended (in which case documenting it in the starter kit would help) or a bug.