There is a bug in flopscope.numpy.linalg.solve that undercounts batched right-hand sides

inside <|Ω|> is LLM generated bug explanation and minimal reproduction.

<|Ω|> When the coefficient matrix is 2-D and the RHS has a leading batch dimension, solve broadcasts the coefficient across the batch and performs one independent solve per batch — but flopscope charges only for a single batch. The cost appears to be derived from the coefficient’s shape and ignores the RHS batch dimension.

Minimal repro (flopscope 0.8.0rc5):
——————————————————————————————————————————————————————————————————————————————————————————
import flopscope as flops, flopscope.numpy as fnp, numpy as np
C = fnp.asarray(np.random.randn(256, 256))
B1 = fnp.asarray(np.random.randn(256, 1024))
B40 = fnp.asarray(np.random.randn(40, 256, 1024))

with flops.BudgetContext(10**15) as b: fnp.linalg.solve(C, B1); print(b.flops_used) # 145,402,538
with flops.BudgetContext(10**15) as b: fnp.linalg.solve(C, B40); print(b.flops_used) # 145,402,538
——————————————————————————————————————————————————————————————————————————————————————————

Both are billed identically, yet B40 does 40× the arithmetic — the 40 batch results are distinct and correct, and wall-clock scales ~40–50× (≈15 ms vs ≈760 ms here). So a batched solve can be charged up to the batch factor too little. This lets a submission run a large deterministic-sample estimator (rewriting each layer A @ W as solve(inv(W.T), A.T) over a batched RHS) while being billed for a small fraction of the real FLOPs. linalg.inv-based batched paths likely share the issue. Flagging as an exploitable measurement gap under the accounting rules. <|Ω|>

6 Likes

Also, the current top submission exploits this hack. Is there a way to retract it or remove it? I couldn’t find one.

3 Likes

Hi @williawa ,

Thanks for the report.
We have replicated this issue, and will be folding in a fix for this into the upcoming release.

Retraction of submissions is not possible, however, once the fix is in, the affected submissions will automatically be evaluated using the patched flopscope.

Thanks,
Mohanty

Update: Fixed in feat(billing): dtype-aware four-factor cost model + reviewer-driven re-tiering by spMohanty · Pull Request #150 · AIcrowd/flopscope · GitHub

6 Likes

When will the fix be merged? It would be nice if it could happen sooner rather than later, because the current state makes it hard to be calibrated about the distribution of scores on the real leaderboard.

Unclear if e.g. 1.5e-7 is a crap score, not even top 20, or if its the best score in the tournament because everything above was gotten through exploits.

10 Likes

this fix has clearly been deployed; big leaderboard upset. looks like not all previous submissions have finished regrading?

@keenanpepper : Yes the re-evals are ongoing as we speak. We will post a formal update in a few hours with more details. Until all the re-evals are done, we expect to have all the new submissions queued up.
We appreciate your patience.

1 Like