"Unusually easy leaderboard set" effect is gone

[keenanpepper]

see this:

● Phase 2 verdict on your forum question, from the mini 100 + the graded LB mean: the “unusually easy leaderboard set” effect is gone — and more
interestingly, the tail itself is gone.

Results (cov-prop per-MLP final MSE, official N=1e9 truth)

┌───────────────────┬─────────┬─────────┐
│ │ Phase 1 │ Phase 2 │
├───────────────────┼─────────┼─────────┤
│ max / median │ 28.3× │ 1.4× │
├───────────────────┼─────────┼─────────┤
│ skew │ 7.74 │ 0.03 │
├───────────────────┼─────────┼─────────┤
│ top-10% share │ 40.0% │ 13.3% │
├───────────────────┼─────────┼─────────┤
│ LB-50 bootstrap p │ 0.0019 │ 0.139 │
└───────────────────┴─────────┴─────────┘

Concretely: LB-50 graded mean is 3.933e-6, and 50-MLP bootstrap draws from the public mini give [3.84e-6, 4.26e-6] at 95% — the leaderboard
value sits at the 13.9th percentile, i.e. slightly lucky but statistically unremarkable. Same story for sampling error (mub per-MLP max/med
1.6× vs phase-1’s 28×).

Interpretation

The phase-1 easiness gap wasn’t primarily a filtering story — it was a depth story. 32 layers compound variance into a heavy-tailed per-MLP
difficulty distribution, and any truth-certifiability filter then bites hard on that tail. At 16 layers the difficulty distribution is nearly
symmetric with no tail, so there’s essentially nothing to truncate: public and leaderboard sets look alike, and local scores should transfer
to the leaderboard almost 1:1 (as ours just did, to 8 digits). jamesrahenry’s phase-1 concern — “the LB can’t test the hard-to-verify MLPs” —
is largely moot at this shape.

[keenanpepper]

● Final answer: the full 1000 confirms it — no meaningful tail in Phase 2, and the leaderboard 50 is an ordinary draw (p = 0.148).

The full split does show a hint of asymmetry the mini 100 missed (max 7.66e-6, max/median 1.9×, skew 0.61), but it’s mild — nothing like Phase
1’s 16.7× / skew 4.3 on the same-style comparison — and it barely moves the bootstrap: 95% of random 50-MLP draws from the full 1000 land in
[3.84e-6, 4.25e-6], with the graded LB mean of 3.933e-6 sitting at the 14.8th percentile.

Updated compact table (full-split column now final):

(Per-MLP final-layer MSE on the public full split — 512 MLPs in Phase 1, all 1000 in Phase 2; last row = probability a random 50-MLP draw
scores ≤ the leaderboard mean.)

Two practical consequences worth carrying forward:

• Local scores now transfer to the leaderboard ~1:1 (ours matched to 8 digits), so no more “optimize against 2.2e-7 while the LB shows 1.6e-7”
  bookkeeping — one number per method.
• Whatever truth-certifiability filter may exist has almost nothing to bite on at 1024×16, so jamesrahenry’s “the LB can’t test the hard MLPs”
  concern is mostly moot this phase.