We are constantly trying to improve this challenge for you and would appreciate any feedback you might have! 
Please reply to this thread with your suggestions and feedback on making the challenge better for you!
All The Best!
Is the smoke test flop budget lower than the full submission budget? I’m seeing submissions run out of budget in the smoke test even though they are under the cap locally, so I suspect it may be half what it should be
Good catch, @jamespayor - confirmed and fixed. The smoke test was running under a stale hardcoded budget (1e10) instead of the real grading budget (6.8e10), so estimators spending anywhere between the two were getting wrongly rejected. The fix is live now (smoke uses the exact same per-MLP budget as grading), and we have re-graded the submissions that were affected, so they should show up scored. Thanks for flagging it!
Affected submissions that were re-evaluated: 309660, 309713, 309714, 309723, 309724, 309738, 309742, 309744, 309745.
Nice, thank you for the fix and rerunning! 
Returning here, it now seems that submissions (mine at least) are getting a much higher walltime penalty than before? Here’s the same code submitted yesterday (AIcrowd | ARC White-Box Estimation Challenge 2026 | Submissions #309886) vs today (AIcrowd | ARC White-Box Estimation Challenge 2026 | Submissions #309977), same flops and walltimes but much increased penalty.
If this is an intended change I’d appreciate clarification; I could be wrong but for instance I didn’t think my code is spending that much time doing things other than dispatching meaty flopscope ops, but I’m now paying a large effective compute penalty.
Hm also fyi on the submissions page the “final layer MSE” column seems to be populated with all-layers MSE scores: AIcrowd | ARC White-Box Estimation Challenge 2026 | Submissions
It seems there has been a rejudge since then, but the current leaderboard is still using pre-rejudge numbers.
Would it be possible to publish the exact flopscope distribution that the grader uses?
I’ve hit some rejections due to the grader’s runtime environment differing from my local env. The starter kit’s pyproject constraint is flopscope>=0.5.0; this resolves to 0.5.0 in the lockfile, but the installed wheel is 0.5.0+np2.2.6, which apparently has a richer API than whatever the grader’s build uses, leading to grader errors like AttributeError: module 'flopscope' has no attribute 'as_symmetric'.
(I also noticed that plain numpy isn’t importable in the grader env—is that by design?)
@omw : The evaluators, as we launch Phase 1, currently use: flopscope v0.8.0rc1 and whestbench v0.12.0rc0.
This might change, as we address the feedback we receive over the next week - (more details here)
For every submission, you can also check out the exact versions that the evaluator uses, in the Configuration tab of the Submission Details page.
(I also noticed that plain numpy isn’t importable in the grader env—is that by design?)
Yes, that is a challenge design choice for whestbench.
We might revisit that decision if we come across strong usecases where using numpy (which counts towards the heavily penalized residual time) is necessary. So we are open to feedback.
Best,
This occurred during an internal update to Flopscope aimed at more accurately distributing time across our three time-accounting buckets. Because the change affected every submission’s score, we re-evaluated all submissions received up to that point.
Hm also fyi on the submissions page the “final layer MSE” column seems to be populated with all-layers MSE scores: AIcrowd | ARC White-Box Estimation Challenge 2026 | Submissions 8
This is fixed now ! Thanks !
Best,