Hi @thylinao,
Yes, we will examine this during the private re-evaluation and prize due diligence.
However, we will not use instrumented share as an automatic eligibility cutoff.
A low instrumented share is a useful signal and may warrant closer review, but it is not conclusive on its own. Some residual time is expected for Python control flow, data preparation, and other supporting code. Numerical Any computational work does not become exempt from compute accounting merely because it runs outside flopscope.
The principle we plan to apply is that participant-side code should not use residual time as a second compute lane that effectively performs more than λ FLOP/s of numerical work while being charged only at λ (= 1e11 FLOP/s). We could of course address this by raising λ globally, but that would disproportionately penalize the large majority of honest submissions whose residual time consists of legitimate supporting work.
If code outside flopscope metering performs, or approximates, large numbers of additions, multiplications, contractions, samples, or similar numerical operations (numerical, boolean or otherwise), we will make an independent effort to estimate the effective computation being performed. There is no single profiler or mechanical threshold that can classify every possible implementation, so this assessment necessarily requires case-by-case technical judgment. The ARC team will make the final determination after reviewing the submission’s implementation and behaviour, and that determination will be final.
To answer your questions directly:
-
A submission is not automatically ineligible merely because most of its charged cost comes from residual time. However, a submission deliberately built around unmetered numerical computation that exceeds what the residual-time charge is intended to represent may be adjusted or re-evaluated. In cases of deliberate circumvention, it may be disqualified.
-
Yes. For submissions in contention for prizes, we will review the instrumented share together with the telemetry and the implementation itself. Optimizing ordinary control flow and data handling is legitimate. Using residual pricing as an alternative arithmetic budget is not.
As mentioned in the Phase 1 update, when we find a general accounting gap, our preference is to correct the accounting and apply the correction consistently to affected submissions. Finding or reporting such a gap is not by itself grounds for disqualification. Deliberately bypassing the accounting mechanism is a separate matter and will affect prize eligibility.
We are also discussing whether Phase 2 can avoid residual-time accounting and λ altogether. Those discussions have not yet resulted in a final design, and we will share the Phase 2 policy once it has been decided. Any such change would apply only to Phase 2 and will not be applied retroactively to Phase 1. We hope this will give us a cleaner long-term solution.
If anyone is unsure whether a particular technique falls on the permitted side of this distinction, please contact us privately at arc-whestbench@aicrowd.com before the deadline.
Best,
Mohanty