I’ve noticed a few submissions on the LB with mean per-MLP compute utilization above 100%. I expected this to trigger the zero-prediction fallback for at least some MLPs, but all public MLPs have similar low final-layer MSEs for those submissions. The adjusted score also appears to use just the uncapped compute multiplier.
Is utilization above 100% intended to be allowed with a proportional score penalty, or should the zero-prediction fallback apply? Could the organizers or anyone please clarify this rule, or correct me if I’m mistaken?
@pranay212 you’re right, and it’s not just a display quirk — I dug into the submission telemetry and your reading is exactly what the evaluator is doing. Posting the measurements here so the observation is pinned to a concrete record, plus a no-auth reproduction, in case it’s useful for the organizers.
This is grader-behaviour only. I’m not naming participants, and I reference one submission ID purely because the claim has to be checkable against a real record.
The check exists in the library but does not fire on the evaluator
whestbench documents and implements a post-hoc combined-budget check: if C_m = F_m + λ·R_m exceeds the per-MLP budget B, predictions for that MLP are zeroed and the multiplier is forced to 1.0.
whestbench 0.14.0, scoring.py:826-843:
combined_budget_exhausted = Falseif ( not budget_exhausted and not time_exhausted and not residual_wall_time_exhausted and is_combined_budget_exhausted( flops_used, residual_wall_time_s, spec.flop_budget, spec.lambda_flops_per_second )): predictions = fnp.zeros((spec.depth, spec.width)) combined_budget_exhausted = True
budget.py:47-51:
def score_multiplier(effective_compute, flop_budget, *, failed): """Per-MLP multiplier: 1.0 on failure (or no budget), else max(0.1, C/B) — uncapped above.""" if failed or flop_budget <= 0: return 1.0 return max(0.1, float(effective_compute) / float(flop_budget))
So a per-MLP multiplier strictly above 1.0 is only reachable when failed is False. If the exhaustion branch had fired, that MLP’s multiplier would be exactly 1.0 and its predictions would be zeros. That gives a clean test of whether the branch ran — which is exactly the “uncapped compute multiplier” you noticed.
What the evaluator actually did
Submission 323861, graded under whestbench 0.14.0 / flopscope 0.10.0, flop_budget = 2.72e11:
| quantity | value |
|---|---|
MLPs with C_m > B (all 100) |
97 / 100 |
MLPs with C_m > B (50 public) |
49 / 50 |
| of those, multiplier > 1.0 | 49 / 49 |
max C_m / B |
1.2327 |
mean_score_multiplier |
1.1243237592904411 |
mean_compute_utilization |
1.1243237592904411 |
n_failed_mlps |
0 |
failure_breakdown.combined_budget_exhausted |
0 |
| median final-layer MSE on the over-budget MLPs | 4.24e-08 |
Two checks confirm your reading:
- The multiplier is
C/B, uncapped. Joiningpublic_scorestoper_mlptelemetry,adjusted_final_layer_score / final_layer_mse == max(0.1, C_m/B)holds on 50/50 public MLPs to a relative tolerance of 1e-6.mean_score_multiplierandmean_compute_utilizationare bit-identical at 1.1243, i.e. the multiplier is just the utilisation, above 1.0. - Predictions were not zeroed. As you noticed, the over-budget MLPs keep low final-layer MSEs — median 4.24e-08. A zeroed prediction scores around 0.9 at this shape, so these are the estimator’s real outputs.
C_m = F_m + 1e11·R_m reproduces effective_compute exactly (max relative error 0.0 across all 100 rows), so λ is 1e11 and the budget is the documented 2.72e11. The inputs to the check are the documented ones; only the branch is missing — which points at the deployed scoring path (schema 2.0, server-side) being a separate implementation from the library that omits the exhaustion branch.
Reproduction (no authentication)
curl -s "https://www.aicrowd.com/challenges/arc-white-box-estimation-challenge-2026/submissions/323861" -o sub.html
import html, jsonB, LAM = 2.72e11, 1e11h = open("sub.html", encoding="utf-8", errors="replace").read()i = h.find(""per_mlp""); start = h.rfind('="', 0, i)seg = h[start + 2:]ev = json.loads(html.unescape(seg[: seg.find('"')]))["props"]["data"]["submission"]["evaluation"]res = ev["results"]tel = {m["mlp_index"]: m["telemetry"] for m in res["per_mlp"]}C = [t["effective_compute"] for t in tel.values()]print("budget :", ev["config_snapshot"]["flop_budget"])print("C > B :", sum(c > B for c in C), "/", len(C))print("max C/B :", max(C) / B)print("failure_breakdown:", res["aggregates"]["public"]["failure_breakdown"])print("mean multiplier :", res["aggregates"]["public"]["mean_score_multiplier"])for e in res["public_scores"][:5]: t, s = tel[e["mlp_index"]], e["scores"] print(f" C/B={t['effective_compute']/B:.4f} applied={s['adjusted_final_layer_score']/s['final_layer_mse']:.4f} mse={s['final_layer_mse']:.3e}")print("max rel err on C=F+1e11*R:", max(abs(t["flops_used"] + LAM*t["residual_wall_time_s"] - t["effective_compute"]) / t["effective_compute"] for t in tel.values()))
Output:
budget : 272000000000.0C > B : 97 / 100max C/B : 1.2327033985992646failure_breakdown: {'budget_exhausted': 0, 'time_exhausted': 0, 'residual_wall_time_exhausted': 0, 'combined_budget_exhausted': 0, 'error': 0}mean multiplier : 1.1243237592904411 C/B=1.0861 applied=1.0861 mse=3.577e-08 C/B=1.1120 applied=1.1120 mse=1.361e-07 C/B=1.0820 applied=1.0820 mse=2.665e-08 C/B=1.0534 applied=1.0534 mse=4.145e-08 C/B=1.1276 applied=1.1276 mse=9.039e-08max rel err on C=F+1e11*R: 0.0
To the organizers, the same question pranay212 asked, sharpened
Is utilization above 100% intended to be scored with the uncapped C/B multiplier, or should the combined_budget_exhausted branch fire (zero predictions, multiplier forced to 1.0)? The library and the scoring-model doc say the latter; the deployed evaluator does the former.
If the cliff is intended, the fix is to wire the existing is_combined_budget_exhausted into the scoring path — no rule change. If a linear C/B penalty above budget is the intended behaviour, that’s a strictly gentler rule than what’s documented and worth stating explicitly in the scoring model, since people are sizing estimators against a hard 2.72e11 cliff that currently isn’t one.
Either way, surfacing C_m/B and the instrumented share on the submission page would let this kind of thing get caught in a day. Happy to re-verify against other submissions if useful.
1 Like
@mohanty , Can you shed some light on this issue please? I see this more prevalent than before on the Leaderboard. And @qi_zhang5 , thanks for a detailed write-up on the issue.
@pranay212 @qi_zhang5 : You’re both right, and thanks for the detailed write-up - it made this quick to confirm.
The combined-budget rule is meant to zero the predictions and force the multiplier to 1.0 when (C_m > B_m), and as you pointed out, while that was correctly working in the local harness, it was not correctly wired on the evaluator.
This was patched yesterday, and has been live on the evaluation servers for some time now. Older phase 1 submissions with C_m > B_m, are being retrospectively patched as well, they should update on the submission page soon.
Thanks,
Mohanty
1 Like