Sharing a footgun that quietly wrecks your score. flops.budget_summary_dict() (and budget_summary()) aggregate over the accumulated per-op history, so calling them inside
predict() — e.g. to read flops_used and self-calibrate your sample count — is very expensive, and that cost lands in residual_wall_time_s, which is billed into effective compute
at λ = 1e11 FLOP/s.
Clean A/B — the SAME estimator, public mini split (100 MLPs, 256×32), local runner, differing only by whether it calls budget_summary_dict() on the per-layer path:
| variant | estimator residual_wall_time_s
|
adjusted_final_layer_score |
|---|---|---|
no budget_summary_dict()
|
0.40 s | 2.03e-5 |
| calls it on the hot path | 62.4 s | 4.60e-5 |
That’s a ~157× residual blowup and a doubled (worse) score from an introspection call that does no estimation work. At λ = 1e11, 62 s of residual ≈ 6.2e12 FLOP-equivalent — ~23× the
whole budget — so on a stricter setup it would zero you outright.
Two things that would help participants:
- A cheap
current_context_flops()accessor: O(1), reads only the active context’s live counter, no history aggregation. - Failing that, a prominent note in the flopscope docs that
budget_summary_dict()/budget_summary()are O(op-history) and must not be called on the hot path.
Workaround we used: call it exactly once per (width, depth, budget) shape to calibrate, cache the per-sample cost, and never touch it again.
Repro: take any estimator, add two budget_summary_dict() calls inside the per-layer loop, run both versions on the mini split, compare the Estimator residual_wall_time_s and
adjusted_final_layer_score.