Flopscope v0.8.0 Release Candidate Available!

ARC White-Box Estimation Challenge 2026
#1

Hi everyone,

We have published the first release candidate of flopscope v0.8.0rc1 to PyPI, paired with whestbench v0.12.0rc0.

This is the version the evaluators will use for Phase 1. We are sharing the release candidate now so you can build against it, check your estimators, and send feedback before the final v0.8.0 release.

TL;DR

  • Install the release candidate:
    pip install --pre "flopscope>=0.8.0rc1" "whestbench>=0.12.0rc0"
  • This is the Phase 1 evaluator version.
  • The cost model is now more consistent: you are billed for computation on values, not data movement.
  • Contraction costs are unified across matmul, dot, inner, outer, tensordot, vdot, einsum, and relevant linalg operations.
  • Residual wall-time accounting is fairer: framework overhead is not charged to your estimator.
  • Weight packing is now pickle-free through flops.Module.
  • Warm-Up evaluators are unchanged and remain on flopscope v0.5.0 / whestbench v0.10.0.

:arrows_counterclockwise: What’s New in flopscope v0.8.0

1. Computation vs data logistics

flopscope now clarifies one core principle:

You are charged for computation on values, not for moving data around.

Arithmetic, reductions, matrix multiplies, transcendentals, and FFTs cost FLOPs.
Copying, reshaping, stacking, concatenating, slicing, and gathering are free.

Matrix operations dominate many estimator budgets and are now counted in full, so please re-check your estimator against the Phase 1 budget.

2. One cost engine for contractions

matmul, dot, inner, outer, tensordot, vdot, einsum, and relevant linalg contractions now share the same symmetry-aware machinery.

This resolves cases where operations such as fnp.tensordot could previously be undercounted. These operations are now billed consistently with the consistent einsum_cost machinery.

3. Fairer residual wall-time accounting

Your score accounts for both FLOPs and residual wall time, which is time spent outside tracked operations.

We re-audited what counts as participant residual time. Framework plumbing, including data transport between the flopscope client and server and array unpacking, is now attributed to flopscope overhead rather than your residual wall time.

In short: you are charged residual time for your own code, not for evaluator plumbing.

4. Pickle-free weight packing and clearer errors

You can now bundle data with your submission through flops.Module. Loading this data is free: it costs 0 FLOPs and is not counted in residual wall time.

The new release also provides clearer errors when an operation is not available in the grading environment.

The full per-operation cost rules are documented in the cost-model.md reference. You can also use budget.summary() to inspect where your own FLOPs are going.

:package: Packing Data Into Your Submission

Define your model as a flops.Module. Array attributes are discovered automatically, and save writes them with a small JSON config instead of using pickle.

# model.py
import flopscope as flops
import flopscope.numpy as fnp

class Linear(flops.Module):
    def __init__(self, n_in, n_out):
        self.W = fnp.zeros((n_out, n_in))     # array state: auto-discovered
        self.b = fnp.zeros(n_out)

    def config(self):                         # non-array config, used to rebuild
        return {"n_in": self.W.shape[1], "n_out": self.W.shape[0]}

    def __call__(self, x):
        return fnp.einsum("oi,i->o", self.W, x) + self.b

if __name__ == "__main__":
    model = Linear(8, 4)
    # ...
    model.save("model.npz")                   # named arrays + JSON config, no pickle

Load the saved module once in your estimator’s setup():

# estimator.py
from pathlib import Path
from whestbench import BaseEstimator
from model import Linear

class Estimator(BaseEstimator):
    def setup(self, ctx):                                                   # runs once
        self.model = Linear.from_file(Path(__file__).parent / "model.npz")  # free load

    def predict(self, mlp, budget):
        ...   # use self.model

The whest CLI bundles everything in your estimator folder, up to 50 MB / 50 files, so model.py and model.npz travel with estimator.py and load for free at grading time.

whest login                                     # once, with your AIcrowd API key
whest submit --estimator estimator.py --watch   # packages, uploads, and follows to a score

Please iterate locally first:

whest run --estimator estimator.py

The full estimator and submission walkthrough is available in the starter kit. The examples/12_save_load_mlp.py example includes a multi-layer flops.Module.

:hammer_and_wrench: Try the Release Candidate

Install the release candidate:

pip install --pre "flopscope>=0.8.0rc1" "whestbench>=0.12.0rc0"
pip show flopscope whestbench   # expect 0.8.0rc1 / 0.12.0rc0

If you are using the starter kit with uv, pin the candidate with:

uv add "flopscope>=0.8.0rc1" "whestbench>=0.12.0rc0"

Then check your estimator against the budget:

import flopscope as flops

with flops.BudgetContext(flop_budget=68_000_000_000) as budget:
    estimator.predict(mlp, budget=68_000_000_000)

print(f"FLOPs used: {budget.flops_used:,}")
print(budget.summary())   # per-operation breakdown

:warning: What Happens if the Cost Model Changes Again?

Phase 1 runs on this release candidate, and it is a candidate for a reason.

If community feedback leads to major changes in the final v0.8.0 release, we will re-evaluate every Phase 1 submission received up to that release on the final cost model. You can submit now without worrying that an early submission will be disadvantaged by a later evaluator change.

:speech_balloon: Send Us Feedback

Please share any issues you find with the updated cost model, flopscope in general, whestbench or the starter-kit; your feedback is extremely valuable and also counts towards the community contribution prizes of 500-5000 USD each!

To avoid any confusion: the Warm-Up evaluators are unchanged and stay on flopscope v0.5.0 / whestbench v0.10.0.

This release candidate is the evaluator version planned for Phase 1, which launches at 00:00 UTC on 18 June 2026 with an independent evaluation environment and a separate leaderboard.

Stay tuned for the official Phase 1 launch announcement, which will cover the updated target architecture, budget changes, leaderboard details, and prize structure.

All the best! :rocket:

:speech_balloon: Feedback & Suggestions
#2
#3

Hi there! I’m encountering a bit of cost-model feedback: currently there are a bunch of savings on the table for manually accounting for input-sparsity (i.e. zeros present in the contraction), pretty relevant because it’s a factor of 2 for passing samples through the net.

Raising this in case you would like to automatically favor/exclude zero entries in matmuls and similar. This is a little realistic for current hardware (at least nvidia GPUs have a sparsity speedup) but idk whether you consider in scope.

#4

Okay a different thing that seems more of an exploit: float32/float64 are billed the same, okay fair enough, but so is complex64. So you can get a “free” factor of 2 flop reduction by packing two values together as a “complex” number and performing float64 * complex64 multiplications.

I think the simple answer here would be to bill real * complex ops as 2x more costly than real * real, complex * complex can be 2x or 4x, or whatever seems appropriate.

#5

@jamespayor : Yes we are aware of this issue, and are currently working on a proper fix for this. The cost adjustments will land in the next release, and affected submissions will be re-evaluated.

We may or may not include the float32/float64 fix in phase 1, but the complex one will definitely be addressed.

1 Like
#6

as discussed in the townhall meeting:
Thanks for the feedback, this is indeed a great suggestion, but we will discuss internally a bit more, if we want to include the sparsity related savings within the scope of Phase 1 and/or Phase 2.
We will share more details once we have a clear decision on this.

1 Like
#7

Thanks! Fwiw I think keeping it uniform across float32 vs float64 is appealing from my perspective, as something that simplifies matters (“just use float64”) though I’d of course be comfortable however y’all decide about it.

#8

James beat me to reporting it but yes, my most recent two submissions are just flops accounting tricks with a “true” score that should be around 2.5e-7. There is some further trickery you can do with quantization and bitpacking to go beyond just the 2x reduction afforded by the complex type. I’m not at my computer now but I’ll reply in a few hours with the details of that.

1 Like
#9

Hello @mohanty ,

Do you have any expectation when the FLOPS estimation for complex number will be fixed? The current situation with the leaderboard, when it is impossible to understand whether it is real scores or hacks with number packing, is confusing a little.

#10

@RomanChernenko : Fair point, and we acknowledge the uncertainty this creates. We are trying our best to address this before the end of this week, but hopefully sooner than that. We will make an announcement as soon as this is resolved.

Best,
Mohanty

1 Like