Submissions close on 17 October 2026 at 23:59 UTC.
Phase 2 changes the rules that govern what your submission code is permitted to do. Code that was allowed in Phase 1 can result in disqualification in Phase 2. Please read Code restrictions in Phase 2 section below before you submit.
Summary of changes
| Item | Phase 1 | Phase 2 |
|---|---|---|
| MLP dimensions | 256 wide, 32 deep | 1,024 wide, 16 deep |
| FLOP budget per MLP | 2.72 × 10¹¹ | 2⁴¹ (approximately 2.199 × 10¹²) |
| Wall-clock cap per MLP | 60 seconds | 120 seconds |
| Computation outside flopscope | Charged against the budget | Prohibited |
| Residual wall time | Priced at the published rate | Capped at 0.4 seconds |
| Submissions per team per day | 50 | 10 |
| Solution process memory | Not separately capped | 8 GB |
| Dataset revision | v1-phase1 |
v2-phase2 |
Accept the updated rules
The official rules were updated for Phase 2.
Before you can make any submissions, you must accept the updated Rules of the challenge.
To review and accept the rules, go to the official rules page.
Code restrictions in Phase 2
The competition measures the accuracy a solution achieves for a fixed amount of computation. That measurement is meaningful only if computation is counted the same way for every participant, and flopscope performs that counting.
In Phase 2, all computation must be performed through flopscope. Your solution’s Python code determines which flopscope operations to call. A submission consists of Python source and data files. At evaluation time, a solution may use only the following:
- The Python interpreter provided by the grader
- The flopscope client API
- Pure-Python standard library modules, for control flow and bookkeeping
Warning: The following are prohibited in Phase 2. This list is not exhaustive. Each entry describes a way of performing computation that flopscope does not count, and any other mechanism having that effect is equally prohibited.
- Bundled or vendored numpy, scipy, or any BLAS or LAPACK implementation
- Compiled kernels of any kind, however delivered, including wheels, shared objects, static binaries, and code generated at runtime
- ctypes, cffi, or any other foreign-function mechanism
- asyncio, threads, subprocesses, multiprocessing, and threading runtimes
- Computation of any kind performed while a flopscope operation is in flight
- Modifying, monkeypatching, or otherwise interfering with the flopscope client, its transport, or its accounting
Automated checks run on every submission. Anything they flag maybe directly invalidated or go through further agent-assisted validation, and anything still unresolved after that is reviewed by a person. Review continues after grading, so a submission can be scored and listed on the leaderboard while its review is still open. Any submission that does not conform to these rules is invalidated once identified.
If you are unsure whether a technique falls inside the rule, ask at arc-whestbench@aicrowd.com before you submit.
Data files remain permitted, including weights, lookup tables, and precomputed artifacts. For how to package and load them, see Ship weights and multi-file submissions in the starter kit. Where the Sponsor considers that a bundled file is not genuinely data, the Sponsor may decline to treat it as data and may disqualify the submission.
How this differs from Phase 1
In Phase 1, a submission could call out to any other library, numerical backend, programming language, executable, or saved file. Computation performed outside flopscope was charged against the budget through the residual wall-time conversion.
That permission does not apply in Phase 2. Residual pricing is removed: the conversion rate λ is 0.0, and residual wall time is capped at 0.4 seconds per MLP instead.
Effective cost is therefore equal to metered FLOPs:
C_m = F_m
Residual time in Phase 2 exists so that your Python code can loop, index, maintain bookkeeping, and determine which flopscope operations to call next. An MLP whose residual time exceeds the cap receives the zero-prediction fallback. It is important to note that this cap is a plumbing allowance, not free compute. It is not an unpriced 0.4 seconds in which to do arithmetic the meter cannot see. Performing meaningful computation in residual time is a breach of the rules and is still disqualifiable, whether or not you stay under the cap. Submissions with unusual residual wall-time usage receive additional scrutiny for conformance to the rules.
If your Phase 1 estimator depends on numpy, a compiled kernel, or a worker thread, it is not eligible in Phase 2. Port it to flopscope operations before you submit.
Model architecture
The best-performing methods in Phase 1 tended to be based on variations of sampling. We have made the network wider and shallower for Phase 2, to make cumulant-propagation-style methods more competitive. Nevertheless, 16 layers is deep enough that we expect handling depth-dependence well to be a challenge.
Phase 1: 256 wide, 32 deep 2.1 M parameters
Phase 2: 1,024 wide, 16 deep 16.8 M parameters
The MLPs are 4 times wider, half as deep, and contain 8 times as many parameters. The FLOP budget increases by approximately the same factor, and the wall-clock cap doubles.
Submission limits
The daily submission cap drops from 50 per team per day in Phase 1 to 10 per team per day in Phase 2. Every submission counts against the same shared team budget, regardless of which team member sent it.
For the first week of Phase 2, failed submissions do not count against the cap, up to 10 additional submissions per day. Use them to get your packaging and submission pipeline working against the Phase 2 evaluator before failures start costing you attempts.
Required software versions
Update all three components before you submit. Do not submit against Phase 1 versions.
pip install --upgrade "flopscope>=0.12.0" "whestbench>=0.16.0"
Pull the latest starter kit. It contains the Phase 2 estimator contract, the per-MLP FLOP budget, and the submission packaging format.
Dataset
Phase 2 uses the v2-phase2 revision of the public dataset on Hugging Face:
load_dataset("aicrowd/arc-whestbench-public-2026", revision="v2-phase2")
Specify the revision explicitly. A cached v1-phase1 copy contains the Phase 1 architecture and is not valid for this round.
Key dates
| Milestone | Date and time |
|---|---|
| Registration and team freeze deadline | 2 October 2026, 23:59 UTC |
| Phase 2 submissions close | 17 October 2026, 23:59 UTC |
| Algorithmic-contribution write-up deadline | 24 October 2026, 23:59 UTC |
Phase 2 carries $100,000 in prizes, in addition to the $50,000 in Phase 1.
Phase 1 status
The Phase 1 private re-evaluations are in progress. We will report the Phase 1 results, the community contribution prizes, and the algorithmic contribution prizes when they are decided.
Send us feedback
Feedback may also count toward the community contribution prizes, which are individual cash awards typically in the range of $500 to $5,000 USD.
- General feedback and discussion: start a topic on the challenge forum.
- Rules questions, FLOP-mispricing reports, and residual-cap exceptions: email arc-whestbench@aicrowd.com. If you believe flopscope is mispricing an operation, report it rather than working around the accounting. Include a minimal snippet, the
flops_usedyou expected, and the value you observed. - Reproducible bugs: open an issue on flopscope, whestbench, or the starter kit.
- Security issues: email arc-whestbench@aicrowd.com privately rather than opening a public issue.