Phase 1 update: flopscope v0.10.0, cost-model fixes, residual-time safeguards, and updated deadlines

ARC White-Box Estimation Challenge 2026
#1

Hi everyone,

Thank you for your patience while we worked on making the challenge fairer and address the issues you raised.

The evaluators now run flopscope v0.10.0 with whestbench v0.14.0.

Below, we explain what the cost model charges, what changed about the compute resources, and how the Phase 1 standings and prizes will be determined.

TL;DR

  • Upgrade:

    pip install -U "flopscope>=0.10.0" "whestbench>=0.14.0"

  • Arithmetic is priced by dtype: the 64-bit class costs 2x the 32-bit class, and complex operations are charged according to the real operations they expand into.

  • Copies, gathers, sorts, and other data-movement operations are no longer free.

  • fnp.einsum(..., out=D) now follows NumPy’s casting rules and raises an error where it previously truncated silently.

  • Participant code now runs on one physical core, while the flopscope backend has access to seven physical cores.

  • Phase 1 closes on 10 August 2026 at 23:59 UTC. The Algorithmic Contribution write-up is due on 17 August 2026 at 23:59 UTC.

  • Prize rankings will come from a private re-evaluation, not the public leaderboard.

  • The registration deadline and team freeze are both now set for 5 September 2026.

:arrows_counterclockwise: What the cost model was giving away

Precision

The meter was dtype-blind, so a float64 multiply cost the same as a float32 multiply. This created a packing channel: two float32 payloads could ride in one float64 lane, producing two results for the price of one. Many submissions exploited this to obtain artificially reduced FLOP counts.

Cost is now scaled by dtype width and complex dtype structure:

charged = int(flop_cost Ă— dtype_rate Ă— complex_factor Ă— weight)

                            before      now
float32 multiply              1          1
float64 multiply              1          2      (dtype_rate)
complex multiply              1          6 real FLOPs, times the width rate
copy, fill, concatenate      free        1 per element written
gather, sort, histogram      free        4 per element written
genuine view, zeros, empty   free        free

NumPy defaults to float64, so if you never considered dtype, your bill has roughly doubled. Casting your hot path to float32 can now produce a real saving.

Implemented in: feat(billing): dtype-aware four-factor cost model + reviewer-driven re-tiering by spMohanty · Pull Request #150 · AIcrowd/flopscope · GitHub

Data movement

In v0.8.0rc0, we made about forty data-movement operations free. The intended rule was that you would be charged for computing on values, but not for moving them around. That rule did not hold up.

Free data movement made it possible to route real computation through a copy chain, such as by precomputing a large table and then gathering from it for free. Many submissions exploited this to obtain artificially reduced FLOP counts.

Implemented in: fix(billing): cost-model accuracy follow-ups from the #150 review by spMohanty · Pull Request #151 · AIcrowd/flopscope · GitHub

Both changes landed in v0.9.0 and v0.9.1, which the evaluators have used since 24 July 2026.

v0.10.0 adds further cost-model changes. A symmetry tag previously kept its discount after asymmetric data had been written into the tagged buffer, and einsum(..., out=D) did not pay for writing D.

Several of these issues and suggestions came from participants.

@williawa found the batched linalg.solve undercount in this bug report, and it was fixed in v0.9.0. @jtel filed both the issue and the fix PR for negative FLOP charges on empty contractions. @ndrew1337 wrote the einsum out= fix, and @andrew_epstein found a billing gap whose fix landed in v0.10.0.

Thank you to them, and to everyone whose report is included in this release without their name attached.

One change breaks code instead of repricing it.

The einsum write-back previously used casting="unsafe" regardless of what you requested, silently truncating values in cases where NumPy would refuse. It now honours your casting= argument, and the traceback identifies the dtypes, the casting rule, and the relevant file and line.

Implemented in: fix(einsum)!: apply the caller's casting rule to out=, as numpy does by spMohanty · Pull Request #168 · AIcrowd/flopscope · GitHub

Because of this repricing, all affected submissions are being regraded. (eval ETA ~8hrs)
The re-evaluation section below has the details.

:desktop_computer: One compute core for participant code

This was a major issue raised by participants over the past two weeks. Resolving it required a significant architectural change on our side.

Residual wall time is converted into FLOPs at a fixed rate, roughly equivalent to one x86 core:

C_m = flops_used + λ · residual_wall_time_s        λ = 1e11 FLOP/s

The participant container previously exposed 16 vCPUs, so one residual second could buy many cores of real computation while being charged at a one-core rate. Some participants exploited this asymmetry between λ and residual wall time to obtain artificially low FLOP counts while effectively running efficient Monte Carlo sampling outside the metered path.

In empirical checks on a randomly sampled subset of submissions, we found that the vast majority of submissions were unaffected: they were flopscope-native and did not use multicore computation beyond what the flopscope-backend supports internally.

Some participants also bundled custom NumPy wheels and compiled BLAS binaries specifically to take advantage of this arbitrage, and circumvent the FLOP counting and take advantage of the assymetric arbitrage.

That is not the intention of the challenge, and we apologize for the confusion it caused among other participants.

We have blocked this for now by limiting participant container code to one CPU core (2vCPUs):

participant CPU     one physical core: 2 vCPUs
flopscope backend   seven physical cores: 14 vCPUs

Residual wall time is intended to cover the Python plumbing around metered operations: the glue that reads the data, arranges it, and hands it to flopscope. It is not intended to provide a second compute lane.

If we find final-evaluation submissions deliberately built around this arbitrage, we will adjust λ as needed to preserve a fair leaderboard. We will continue monitoring submissions until the end of Phase 1.

For Phase 2, we are discussing whether to require all numerical work to run exclusively through flopscope. No decision has been made. We will share the final policy with the Phase 2 announcement.

:calendar: Phase 1 deadline extended to 10 August, Registration deadline extended to 5th September, and how the final standings and prizes will be decided

Phase 1 submission deadline              10 August 2026, 23:59 UTC
Algorithmic Contribution write-up        17 August 2026, 23:59 UTC

Registration deadline                     5 September 2026, 23:59 UTC
Team freeze deadline                      5 September 2026, 23:59 UTC

So far, the following has only appeared in a thread reply, so we want to state it clearly here.

All Phase 1 submissions are evaluated on 50 public and 50 private MLPs. Each team may select up to two submissions for the Phase 1 private re-evaluation. If you select none, we will use your two highest-ranked submissions on the Phase 1 public leaderboard.

That re-evaluation will run on a completely separate, freshly generated test suite of randomly initialised MLPs, using private seeds that were not used during either Phase 1 or Phase 2.

The suite will follow the same distribution, with the same width and layer-count ranges and the same FLOP-budget calibration, but the MLP instances will be entirely different. We will choose the test-suite size using appropriate statistical analyses so that we can reliably distinguish between closely performing solutions.

Some submissions were clearly overfitting to the public MLPs, including their IDs and names. This is allowed, but the resulting performance is unlikely to generalize to the post-Phase 1 private re-evaluation.

public MLPs, Phase 1                     50
private MLPs, Phase 1                    50
Phase 1 private re-evaluation suite      freshly generated, private seeds
                                          same width, depth, budget calibration
                                          size chosen by statistical analysis
submissions each team may nominate       up to 2
if you nominate none                     your top 2 on the public board

Prize rankings will be determined exclusively by the post-Phase 1 private re-evaluation results, not by any score displayed on the public leaderboard during the competition.

A narrow lead on the public leaderboard is therefore not a final result, and there is nothing to be gained by tuning specifically to the 50 public MLPs, as some teams are currently focussing on.

We will email each team with instructions for nominating submissions. In the meantime, please keep a note of the submission IDs you intend to nominate.

To avoid confusion, the following have not changed: the scoring formula, λ, the per-MLP FLOP budget, the 0.1 multiplier floor, and your permission to bundle your own libraries, native code, and precompiled artifacts.

What changed is how operations are priced and how much hardware participant code can use.

We retain the right to adjust λ where necessary, particularly if participants intentionally try to exploit the available residual wall time.

During final evaluation, we may also make targeted changes to the cost model if we identify an exploit that creates a material gap between the charged cost and what the computation would cost in practice. Finding such an exploit will not, by itself, result in disqualification. We will patch the gap, apply the revised rule consistently to all affected submissions, and rerun them as needed, and communicate the decisions transparently with all the participants.

For Phase 2, we may cap the residual wall time available per MLP for all submissions. This is still under discussion, and we welcome your feedback.

:speech_balloon: Send us feedback

Please share any issues you find with the updated cost model, flopscope, whestbench, or the starter kit. Your feedback is extremely valuable and also counts towards the community contribution prizes of USD 500–5,000 each.

The starter kit already uses the new versions, so running uv sync will give you a matching environment. The per-operation rules are documented in cost-model.md, and budget.summary() shows how these changes affect your own estimator.

All the best! :rocket:
Mohanty

2 Likes
Rules clarification: are operations on the NumPy-backed arrays reachable through flopscope.numpy inside the intended accounting boundary?
#2
#3
#4

Thanks for the detailed write-up and for the extension. Four pieces of feedback from re-deriving the estimator against 0.10.0, three measured and one a smaller modelling point. Everything below is reproducible from the public repos and the public submission pages.

1. The op-weight and dtype-rate tables are byte-identical between 0.9.1 and 0.10.0

The “Before / Now” table reads as though copy/fill/concatenate went free to 1 per element, and gather/sort went free to 4 per element, in this release. For anyone who was already on 0.9.1, which per the RUNTIME ENVIRONMENT panel on our graded submissions is what the evaluators were running before this update, nothing changed at all:

$ git -C flopscope checkout v0.9.1    # 63a6121c "bump: version 0.9.0 -> 0.9.1"$ shasum -a 256 src/flopscope/data/default_weights.json9ff1647a0048d2bd23a7a3d76ee0c60bfd3670d03b15ad8bf2b911c2ae19539f$ git -C flopscope checkout c0e62e14  # "bump: version 0.9.1 -> 0.10.0"$ shasum -a 256 src/flopscope/data/default_weights.json9ff1647a0048d2bd23a7a3d76ee0c60bfd3670d03b15ad8bf2b911c2ae19539f

Same hash: 472 op weights and 18 dtype rates, zero diffs. We had independently measured on 0.9.1 that concatenate = 1.0 per output element, take = 4.0 per output element and reshape = 1.0 per element, so those prices were already in force.

The table appears to be measured against a pre-0.9 baseline. That is worth stating explicitly in the post, because as written it sends people re-auditing code for a repricing that did not happen on the version they were actually graded on. (The einsum(..., out=) casting change is real and behavioural, that part was clear.)

2. The documented wall limits do not match deployed behaviour

ContestSpec in the pinned whestbench declares wall_time_limit_s = 60 and predict_timeout_s = 30, and runner.py calls _terminate_process() on predict-timeout expiry without restarting, which reads as “one slow MLP zeroes every later MLP on that worker”.

Currently-graded submissions with 0 failures out of 50 show per-MLP walls well past both numbers. We can see 47 s, 55.8 s and 64.8 s on the public per-MLP ledgers, the last of which is above the documented 60 s hard limit.

Could you confirm the values actually deployed, and what window the timeout is measured over? We sized several approaches against a 30 s fatal ceiling and rejected them. If the real envelope is 60 s or more of wall, that changes what is worth building, and I doubt we are the only team that read the constants literally.

3. The residual-time safeguard reduces the arbitrage but does not close it

The one-core change clearly bit, and it moved the top of the leaderboard by roughly 6x. But post-update, some graded submissions still report an instrumented share of about 1e-5: analytic F of 2e6 to 1.3e7 per MLP against C of 2.3e11 to 2.7e11, meaning essentially all of the charged compute arrives through the residual term. The distinguishing signature on the public ledgers is that F is identical to three significant figures across all 50 MLPs. No sampler produces a constant F, because pruning and any adaptive allocation are data-dependent (ours varies 1.14x across nets, and one honest entry varies 4.7x).

The arithmetic for why lambda still under-prices it: at lambda = 1e11 FLOP/s, one second of residual buys 1e11 charged FLOPs. A single modern core running well-optimised dense float32 GEMM does roughly 1.5e11 to 2e11 FLOP/s (2 FMA ports x 16 float32 lanes x 2 flops x ~3 GHz). So residual remains cheaper than instrumented arithmetic by roughly 2x for exactly the workload this challenge is made of, and the incentive to move work off the instrumented path survives at reduced strength.

Three options that would close it:

  • Calibrate lambda to measured single-core peak for the dominant dtype, with a margin, i.e. raise lambda rather than restrict cores further. Cores are easy to restrict and easy to work around. The price is the actual control.
  • Cap the residual share , e.g. R * lambda <= k * F, or floor the multiplier when the instrumented share drops below a threshold. This targets the signature directly and is a no-op for anyone whose share is near 1.
  • Publish the instrumented share on the submission page next to BUDGET USED. It costs nothing, it is already computed, and simply making F/C visible per submission would let the community self-police rather than putting the whole burden on the private re-evaluation.

Happy to share exact submission IDs privately if useful. They are visible on the public leaderboard, so I would rather not single people out in a public thread.

4. Minor: float16 bills at rate 1.0, the same as float32

dtype_rates has float64: 2.0 and float32: 1.0, but also float16: 1.0, int8: 1.0, int16: 1.0. If the principle is that the 64-bit class costs 2x the 32-bit class because the arithmetic genuinely costs more, then the symmetric case is that fp16/bf16 arithmetic genuinely costs about half on hardware that supports it, so a rate of 0.5 would be the consistent choice. As it stands the model over-charges narrow dtypes relative to its own stated principle.

This one cuts against participants rather than for them, so it is not urgent. But it does mean the cost model currently gives no reason to ever use a narrow dtype, which is an odd signal for a challenge about compute efficiency.

Thanks again. The per-MLP ledger on the submission pages is genuinely excellent, and the vs Monte Carlo column in particular made it much easier to reason about where our compute was going.