Rules clarification: are operations on the NumPy-backed arrays reachable through flopscope.numpy inside the intended accounting boundary?

ARC White-Box Estimation Challenge 2026
,
#1

Threads 18099 and 18108 have already described, publicly and with measurements, that arrays reachable through flopscope.numpy can be operated on without those operations being registered as instrumented FLOPs, so the work is charged only through the residual wall-clock term at 1e11 FLOP/s. I am not adding anything new here or asking about any particular submission but I was wondering where the boundary is, because it seems like two parts of the rules point in different directions.

Section 5.2 and section 5.6 state that bundling native code is permitted, and that libraries which are not pre-installed may be shipped and are charged through wall time. Read on its own, that suggests doing real arithmetic outside the instrumented path is an intended and priced tradeoff.

Section 12 lists “attempts to evade flopscope accounting” as grounds for disqualification. The challenge announcement gives as its example of unambiguous hacking “modifying constants or counts held in memory.” The behaviour above does not modify any count. It never reaches the counter at all.

So my question is:

  1. Is arithmetic performed on an array obtained from flopscope.numpy, where that arithmetic does not pass through the instrumented path and is charged only via residual wall time, inside or outside the intended accounting boundary?
  2. Does the answer change if the array is obtained deliberately rather than incidentally?
  3. Does the answer change between Phase 1 and Phase 2, given that flopscope 0.10.0 shipped on 30 July and section 5.3 permits re-grading under a corrected grader?
    Thank you!
8 Likes
#2

Want to +1 this as clarity here would be appreciated. At the moment, the public leaderboard appears to be significantly determined by who can build the fastest native backend for uninstrumented, residual billed operations.

EDIT: Given the impending Phase 1 deadline, this ambiguity has immediate competitive consequences. Stopping work on native optimization risks falling further behind if the accounting remains unchanged, while continuing risks substantial wasted effort if the approach is later prohibited or repriced.

3 Likes
#3

I agree. The 1 week extension is helpful, but without a clear announcement regarding the final pricing mechanism and structure, we’re stuck in the same spot as last week.

6 Likes
#4

Yes, it seems like even if arithmetic performed on an flopscope array being done through residual time is allowed, and this isn’t changed, at least capping the residual time in some manner would be a meaningful change in terms of making the leaderboard accurately reflect rankings.

This could be pretty basic, such as pinning residual time to be under a certain percentage of the total time utilization (which seems like for basically every estimator that isn’t using the residual time exploit, is 1% or less of their total estimator runtime), or just capping residual time flops through only allowing it to be up to a certain what percentage of the non residual time flops, and all submissions above that are invalid or penalized much more heavily scorewise.

3 Likes
#5

the people need answers

#6

We have posted the details to all the questions on this post:

@thylinao

  1. Is arithmetic performed on an array obtained from flopscope.numpy, where that arithmetic does not pass through the instrumented path and is charged only via residual wall time, inside or outside the intended accounting boundary?

on the eval servers you have flopscope-client and not the flopscope core package, so you wont be able to do operations directly on the array that bypass the accounting boundary. You can ofcourse bundle in your own numpy wheel, but with the recent changes, and the fact that you will have 1 core available to you, it would be cheaper for you to use flopscope instead for the same computations.
to illustrate how flopscope-client behaves on the eval servers:

python3 -m venv /tmp/fs && /tmp/fs/bin/pip install -q flopscope-client && /tmp/fs/bin/python - <<'EOF'
import flopscope.numpy as fnp

print("what fnp gives you :", fnp.ndarray.__name__)
print("base classes       :", [c.__name__ for c in fnp.ndarray.__mro__])
print("what it stores     :", fnp.ndarray.__slots__)

a = fnp.ndarray(None, (4, 4), "float64")   # the object fnp.zeros((4,4)) hands you
a.base                                     # the "raw NumPy array" the claim relies on
EOF

what fnp gives you : RemoteArray
base classes       : ['RemoteArray', 'object']
what it stores     : ('_handle_id', '_shape', '_dtype', '_symmetry', '__weakref__')
Traceback (most recent call last):
  File "<stdin>", line 8, in <module>
AttributeError: 'RemoteArray' object has no attribute 'base'

Does the answer change between Phase 1 and Phase 2, given that flopscope 0.10.0 shipped on 30 July and section 5.3 permits re-grading under a corrected grader?

As described in the attached post, after Phase 1 ends, we will have a dedicated private re-evaluation of the top-submissions, before the final Phase 1 leaderboard will be decided. This is similar to Phase 2, but given the number of adversarial submissions we have already received in Phase 1, we believe it would be fair to more thoroughly assess the top submissions before any leaderboard positions (and prizes) are assigned.

1 Like