How the hidden-set evaluation invokes whest run

ARC White-Box Estimation Challenge 2026
1

Hi,

The package applies memory_limit_mb as an RLIMIT_AS only on the --runner subprocess path; on the default --runner local the limit is advisory and the package logs that it is not enforced. Could you confirm which runner the official evaluation uses, and which memory_limit_mb value applies? The constant differs between releases (65536 in 0.14.0, 8192 in 0.16.0), so the answer changes which of our local measurements are meaningful.

Thanks.

2

Following up, and narrowing this: rules section 5.6 states the 8 GB limit for Phase 2, thank you, so the “which value” half is answered. Two details remain:

1. Is the limit applied to virtual address space (RLIMIT_AS / VmPeak) or to resident memory?

2. Does one Solution process persist across all MLPs in a suite, including an expanded suite drawn under section 5.4 or the section 6 tiebreaker, or is it restarted between MLPs or batches?

Thanks.

3

One further detail on long-lived workers. In whestbench 0.16.0 each predict call runs inside a fresh flops.BudgetContext, and on exit flopscope 0.12.0 appends that context’s full op log to its session-wide accumulator, which the worker never clears. With many MLPs in one worker, the process therefore grows with every MLP under the 8 GB limit.

May a Solution call the public flopscope.budget_reset() as the first statement of predict? It does not change the active context’s flops_used or budget, and the per-MLP budget breakdown is unchanged. Alternatively, could the harness clear the accumulator between MLPs?

Thanks.

4

In the grader’s client/server mode the participant process runs flopscope-client 0.12.1 (_remote_array.py, _connection.py and so on), which keeps no op log: no _op_log and no BudgetAccumulator anywhere in that package.

The accumulator you describe is in the full flopscope package that flopscope-server runs. In flopscope/_budget.py, BudgetAccumulator.record() is called from every BudgetContext.__exit__ and appends a NamespaceRecord holding a copy of that context’s op log (list(self._op_log[start:])). Only budget_reset() clears it. On the server that reset is grader-only (budget_summary_reset in flopscope_server/_server.py needs the control token), and the client’s own budget_reset() only resets local counters on the active context.

whestbench/subprocess_worker.py applies RLIMIT_AS to the participant worker, so per-MLP growth from it would show up in the server process, not in the capped participant process. Whether that server process persists across MLPs on the official grader, and what limits it has, only the organisers can say. Could an organiser confirm?

One small data point from a local replica of this mode: three predicts of a roughly 50,000-call estimator in one persistent client process gave client RSS of 21-27 MB and server RSS of 2.1-4.0 GB, with no visible growth. That is far too few to rule out growth over hundreds of MLPs.

5

We reproduced this memory growth on the in-process and flopscope-server evaluation paths in flopscope 0.12.3, with standalone reproductions and measurements in `BudgetAccumulator.record()` keeps every closed context's raw op log for the life of the process, so memory grows with each scored MLP · Issue #321 · AIcrowd/flopscope · GitHub. Each closed BudgetContext appends its diagnostic_ops list to the BudgetAccumulator. Across five 1024x16 MLPs, 504aldo’s V16b retains 18,970 OpRecord entries and adds 1.95 MiB heap per MLP, while a 50,000-op estimator adds 19–25 MiB per MLP. A one-line record() change gated by retain_op_log drops retained heap across eight 15,000-op contexts from 44.905 MiB to 0.045 MiB with identical flops_used; with the module test fixture enabling retain_op_log, the scan-oracle checks pass and the full 10,702-test suite passes.

[Phase 2 write-up] A cost-engineered factorized K=3 chain + transferred LT2 corrector (3.82e-9) (Submission #337389)
6

Adding a related question about the 120 s per-MLP wall-clock limit (rules §5.6), since it sits next to the memory question above. I couldn’t find what happens to a single MLP that goes over it. Could an organiser clarify:

  1. What the limit covers. Is it measured on the predict call only, or does it include setup, process start-up and the client/server round trips of that MLP?
  2. What a timed-out MLP scores. Is it marked failed and given a fixed penalty (and if so, which one: the sampling baseline, a zero prediction, something else), excluded from the mean, or does one timeout fail the whole submission?
  3. Retries. Is a timed-out MLP retried, or is the first attempt final?
  4. Hidden-set grading. Does the final private evaluation run on the same hardware with the same limit as the public grader?

For context: grading an identical package twice, we saw the same MLP’s wall time differ by up to ~11 s between runs, so the answers to 1–3 decide how much margin is needed.

Thanks!