The mechanism
The per-MLP cost is
C = instrumented_FLOPs + λ · residual_wall_seconds, λ = 1e11 /s
The residual term prices un-instrumentable overhead a submission unavoidably incurs (Python glue, framework cost). It has also become a primary compute channel: a submission can do much of its real arithmetic in bundled native code, which flopscope doesn’t count, and pay for it through the wall-time conversion at a rate that sits below the throughput of modern AVX-512/VNNI silicon. As a result, standing among the top submissions reflects kernel-level engineering and grading-host scheduling alongside estimation quality.
Two observations point the same way:
- top submissions show instrumented/effective-compute ratios below 0.001 (most of the cost is wall-time rather than counted operations);
- identical submitted bytes score quite differently across grading windows, since wall-time depends on fleet load and the CPU affinity of the assigned worker slot — a source of variance that also makes a faithful re-grade harder.
Both share the same root: the score depends in part on hardware and scheduling, not only on the algorithm. Bundling native code and paying for it through the wall-time conversion is expressly permitted (§5.2, §5.6), but it does undermine what the competition set out to measure. (The rules already intend that conversion to be charged at an “unfavorable” rate, §5.5–5.6; the practical issue is that on current vector hardware it is not.)
The principle for the fix
In a FLOP-budgeted estimation challenge, the score should be a function of the algorithm, not the hardware. Analytical FLOP count is an algorithm property (a matmul is the same number of FLOPs on any machine); wall-clock time is a hardware property. Any cost channel that prices real computation by wall-time reintroduces hardware dependence and can be arbitraged.
Three fixes, surgical → structural
Option A — cap the residual channel
Keep the formula, but bound the wall-time contribution so it can only ever absorb genuine overhead, never serve as a compute channel:
C = instrumented_FLOPs + λ · min(residual_seconds, τ)
with τ a small fixed allowance (enough for real glue, e.g. tens of ms), or equivalently flag/reject any submission where λ·residual > α · instrumented_FLOPs for small α (≈0.05–0.10). Rationale: a submission whose residual dominates is doing substantial uninstrumented computation and is out of spec. The smallest change to the existing formula; the weakness is that τ/α need tuning and a little hidden compute can still hide under the cap.
Option B — re-price the residual to the true hardware rate
Raise λ to the grading host’s actual sustained vectorized throughput (measure the grader’s peak AVX-512/VNNI FMA rate for the relevant dtypes; this is several ×1e11 to ~1e12+). Then a native FLOP costs approximately what an instrumented FLOP costs, and the arbitrage disappears. Simple, but imperfect: a single λ cannot match every op’s true rate (a dense matmul and a transcendental differ by ~100×), it is silicon-specific, and the score stays hardware-dependent — so the re-grade nondeterminism persists. Best used as a complement to A, not alone.
Option C — score on analytical compute only
Remove wall-time from the score entirely and forbid uninstrumented computation:
C = instrumented_FLOPs (wall-clock kept ONLY as a generous anti-DoS timeout,
never as a score input)
with the requirement that all computation is accounted through the counted API — work the meter cannot see is either rejected or required to declare a verifiable analytical FLOP count, detected via the instrumented/effective ratio.
This does three things at once:
- Kills the CPU-optimization game — a matmul costs its FLOPs whether it runs in 1 ms or 1 s, so hand-tuned kernels and hardware speed buy nothing.
- Makes the score hardware-independent and deterministic — identical bytes score identically on any machine, at any fleet load. This simultaneously cures the re-grade instability and the window-to-window variance. One change, two problems.
- Restores the intended contest — the only lever left on cost is a better algorithm (fewer FLOPs for the same accuracy), which is exactly white-box estimation.
Honest tradeoff: Option C restricts the bundled-native-code allowance (rules 5.2/5.6) to code that reports counted FLOPs. It is the largest change, but the only one that fully restores the estimation framing.
These are offered as options rather than a single prescription — the right balance between minimal disruption and fully restoring the estimation framing is a call for the organizers, and it depends on constraints we can’t see.
Disclosure
This note was drafted with the assistance of an AI system and reviewed before sending. The analysis reflects our own experimentation against the official scoring pipeline; any errors in it are ours.