It’s kind of baffling because, AFAIK, the organizers have not informed us of any selection process that would make the public 50 lack this tail of high-variance MLPs. And yet, if it were really sampled from the same distribution as the huggingface datasets, there would only be a 0.2% chance of it being this easy. Which seems to strongly imply there was some kind of filtering that was used… perhaps rejection sampling?
So a question on my mind is, will the private 50 also lack this tail of difficult / high-variance MLPs, or will it contain the tail?
I agree. We had the same observation earlier that our local score taking from the public dataset has been consistently worse than what we saw from the leaderboard.
They’ve been very clear the whole way through that the evaluations will be done on separate MLPs, but they don’t say how those MLPs are chosen.
The Private Re-evaluation test suite is drawn from the same distribution as the per-Phase suites (same width and layer-count ranges, same FLOP-budget calibration) but uses different random MLPs, so Solutions that overfit to specific MLPs or specific seeds will be penalized accordingly.
If leading submissions remain statistically indistinguishable after the Private Re-evaluation, Sponsor may generate additional MLPs from the same distribution to reduce uncertainty, per the tie-breaking procedure in §6."
Keenan, I think it’s awesome you’ve spent the time to dig into this, and I think it highly unlikely that this wasn’t deliberate. The only way I can think this would happen is that these MLPs were sorted somehow and the first 100 were chosen for Phase 1 public evaluations.
Maybe that’s Phase 2? 1000 MLPs with the hard tails included?
There is a benign filter that explains the data without rejection-sampling-for-easiness: the grader needs its own Monte-Carlo ground truth per MLP, and high-variance MLPs are exactly the ones whose MC truth is expensive or uncertifiable. If the organizers kept only truth-certifiable MLPs, that filter truncates the variance tail — and it would apply to the private 50 as well, since manual validation needs reliable truth even more. So the private set may look like the LB 50, not like the full split.
If pluto’s right, that’s a bit of a problem: it means the truth-certifiability filter and “is this net mechanistically interesting” aren’t independent. A net whose MC ground truth is expensive or uncertifiable is usually expensive/uncertifiable because something in its structure produces high-variance or heavy-tailed output — which is exactly the kind of thing white-box access should give you an edge on finding. If that filter runs on both the public and private sets, we’re not doing white-box estimation on the hard-to-white-box MLPs — we’re doing it on the MLPs that were already easy enough for black-box MC to nail down in the first place.
Put differently: the grader can only score what it can verify, so whatever’s hardest to verify never gets tested, regardless of how well any given method would actually do on it. That’s not a knock on the challenge design — everyone needs trustworthy ground truth to grade against — but it might mean the leaderboard (public or private) can’t actually tell us whether anyone’s exploiting weight access on the cases where exploiting it would matter most.