You are supposed to use flopscope.numpy for calculations inside submissions. When uploading an entry here, the system correctly blocks it by forbidding the import of numpy.
However, flopscope.numpy still exposes NumPy-backed arrays without explicitly importing NumPy.
Operations on that raw array perform real computation but report zero instrumented FLOPs, leaving the work graded only through the residual wall-clock penalty.
Because this uses a fixed wall-clock rate, multicore or really fast execution may receive substantially more real computation for the same charged compute.
This can increase someone’s score by more than 10x depending on the hardware of the machine where the grader runs.
I reproduced this locally with the contest-matched FlopScope version, but I have not submitted an entry using it. I can provide organizers with a minimal reproduction.
Has this path already been patched or explicitly ruled permissible? In particular, does the grader prevent raw-array operations obtained without an explicit import numpy, or does it only gate the import
itself?
Most crucially this would not be catched by overfitting corrections or the sealed nets at the end of phase 2.
A clarification would help me know if the leaderboard is accurate, as right now, a rank 60 or rank 20 submission may be actually in the top 3 without any way of knowing.
To your question about whether the leaderboard is accurate—this can actually be checked using public data, and based on what the data show, this path does appear to be in use.
Each submission’s evaluation data includes per-MLP flops_used and residual_wall_time_s, as well as mean_effective_compute. Computing the following simple ratio clearly separates the two cases:
For my own entry (319062), the ratio is 0.93 —nearly all of the charged compute went through the instrumented path. For several submissions currently near the top of the leaderboard, this value is below 0.001 , meaning that almost all of their charged compute comes from residual wall-clock time.
The figures needed to calculate this ratio are visible on every submission page: MEAN EFFECTIVE COMPUTE in the “Compute composition” panel, and the per-MLP FLOPS column in the ledger table at the bottom.
I think this supports the point you raised. Under the current mechanism, the actual amount of computation is under-reported, so the score no longer represents what the benchmark is intended to measure—accuracy per unit of compute.
As a result, two submissions can differ by an order of magnitude in score even when the quality of the underlying estimator is identical. As you noted, re-evaluating the submissions on a fresh private suite would not resolve this issue, because the behaviour causing the under-reporting is contained in the submitted code itself.
Based solely on the currently available public data, someone viewing the leaderboard cannot distinguish between the two cases. That is the part I think is worth fixing.
We reproduced the local behaviour you described. In the full flopscope package, FlopscopeArray is a NumPy ndarray subclass. It can be converted into a plain ndarray, and computation on that ndarray adds no instrumented FLOPs. That work is charged through residual wall time.
The production grader uses a different execution path. Participant code runs against flopscope-client, and arrays in the participant process are RemoteArray proxies for data held by the trusted flopscope-server. The grader does not run submissions through any of the whest --runner modes. The inherited ndarray routes from the local package, including .view(np.ndarray) and the ndarray buffer interface, therefore do not expose the server’s underlying array.
Participants may currently ship their own NumPy, BLAS, or native numerical code. Computation performed there is not included in flops_used; it is charged through residual_wall_time_s. We therefore should not describe the client/server setup as making all off-meter numerical work impossible. It prevents the specific inherited-ndarray escape from the full local package.
As of 3 August, participant code runs on one physical core. This reduces the multicore arbitrage such that existed when residual compute could use many cores while being charged at a fixed per-second rate. For most operations, the residual time cost of doing the operation outside flopscope is higher than doing directly on flopscope. The exceptions are limited to matmuls of specific sizes where the compute intensity of numpy is highest.
For Phase 1, prize rankings will come from the private re-evaluation on the updated environment with the one-core limit. For Phase 2, we are discussing whether to require all numerical work to go through flopscope.
Thank you again for reporting this. The local behaviour is real, and your telemetry analysis helped separate the specific array escape from the broader residual-compute question.
