The Phase 1 update asked for feedback on whether to cap residual wall time per MLP in Phase 2.
We measured that trade-off fairly carefully while building our Phase 1 entry, so here is the
data rather than an opinion. All of it is from our own submissions, on one machine, with
flopscope 0.10.0.
THE MEASUREMENT
Strassen-Winograd matrix multiplication is a clean test case, because recursion depth trades
metered FLOPs against Python-level call count almost linearly. One layer at width 256, chunk
65,536, measured through the meter itself:
depth metered FLOPs/sample residual-equivalent/sample TOTAL
0 130,816 142 130,958
1 114,945 717 115,662
2 101,507 2,662 104,169 ← optimum
3 90,534 22,124 112,658
4 82,307 74,196 156,503
5 77,515 432,427 509,941
“Residual-equivalent” is residual_wall_time_s x 1e11, i.e. the units the score already charges
it in. Depth 5 saves 24,000 metered FLOPs per sample and spends 432,000 residual ones — an 18:1
loss. We confirmed the same shape at chunk 8,192, where the loss at depth 5 is 133:1.
WHAT WE TAKE FROM IT
-
For algorithmic uses, lambda = 1e11 already works as a disincentive. It is not a loophole
here: it made an asymptotically faster matrix multiply strictly worse past depth 2, and we
shipped depth 2 because of it. Whatever a cap is for, it is not needed to stop this. -
The optimum is implementation-dependent and moves with chunk size. That is worth knowing
before setting a number, because a cap does not just forbid a range — it changes which
algorithm is optimal, and it does so differently for different code. -
For calibration, our own submissions run at 5-46 ms of residual per MLP (0.2-1.7% of budget)
with everything inside flopscope primitives. So a cap anywhere at or above ~250 ms per MLP
would leave an estimator like ours a 5-50x margin and, as far as we can tell, would not
penalise ordinary implementation overhead. We offer that only as one data point for where the
floor is; we have no view on where the ceiling should be. -
If a cap is introduced, we would gently suggest announcing the exact accounting alongside it
— specifically whether numpy time inside a counted primitive is residual (it is not, today:
residual = wall - flopscope backend - flopscope overhead). We got that wrong ourselves at one
point and it cost us a day, and it is the sort of thing that is much cheaper to state than to
rediscover.
ONE UNRELATED METERING FACT, since it may save someone else the experiment
flopscope prices a batched matmul at exactly B x (2m-1) x m x c — i.e. per product, with no
discount and no penalty for batching. Measured on (49, 64, 64) @ (49, 64, 8192) and
(343, 32, 32) @ (343, 32, 8192): both match the closed form to the FLOP. So restructuring many
small products into one batched call changes wall time but not the meter.
CAVEATS
One machine, one estimator family, flopscope 0.10.0. The residual column in particular is
machine- and load-dependent — we have seen the depth-2 vs depth-3 ordering flip under contention
— so treat the totals as indicative and the metered column as exact.
Happy to share more detail if it is useful. Our Phase 1 code and full research log, including the
measurements above, are public: GitHub - Oishi1029/arc-whestbench-2026: Phase 1 solution and research log for the ARC White-Box Estimation Challenge 2026 · GitHub