How the hidden-set evaluation invokes whest run

ARC White-Box Estimation Challenge 2026
1

Hi,

The package applies memory_limit_mb as an RLIMIT_AS only on the --runner subprocess path; on the default --runner local the limit is advisory and the package logs that it is not enforced. Could you confirm which runner the official evaluation uses, and which memory_limit_mb value applies? The constant differs between releases (65536 in 0.14.0, 8192 in 0.16.0), so the answer changes which of our local measurements are meaningful.

Thanks.

2

Following up, and narrowing this: rules section 5.6 states the 8 GB limit for Phase 2, thank you, so the “which value” half is answered. Two details remain:

1. Is the limit applied to virtual address space (RLIMIT_AS / VmPeak) or to resident memory?

2. Does one Solution process persist across all MLPs in a suite, including an expanded suite drawn under section 5.4 or the section 6 tiebreaker, or is it restarted between MLPs or batches?

Thanks.

3

One further detail on long-lived workers. In whestbench 0.16.0 each predict call runs inside a fresh flops.BudgetContext, and on exit flopscope 0.12.0 appends that context’s full op log to its session-wide accumulator, which the worker never clears. With many MLPs in one worker, the process therefore grows with every MLP under the 8 GB limit.

May a Solution call the public flopscope.budget_reset() as the first statement of predict? It does not change the active context’s flops_used or budget, and the per-MLP budget breakdown is unchanged. Alternatively, could the harness clear the accumulator between MLPs?

Thanks.