They’ve been very clear the whole way through that the evaluations will be done on separate MLPs, but they don’t say how those MLPs are chosen.
The Private Re-evaluation test suite is drawn from the same distribution as the per-Phase suites (same width and layer-count ranges, same FLOP-budget calibration) but uses different random MLPs, so Solutions that overfit to specific MLPs or specific seeds will be penalized accordingly.
If leading submissions remain statistically indistinguishable after the Private Re-evaluation, Sponsor may generate additional MLPs from the same distribution to reduce uncertainty, per the tie-breaking procedure in §6."
Keenan, I think it’s awesome you’ve spent the time to dig into this, and I think it highly unlikely that this wasn’t deliberate. The only way I can think this would happen is that these MLPs were sorted somehow and the first 100 were chosen for Phase 1 public evaluations.
Maybe that’s Phase 2? 1000 MLPs with the hard tails included?