[Phase 1 research atlas] How I reached public #16 — and why the next 56 experiments failed

At one point in Phase 1, my earlier estimator reached public rank #16 under
the then-active evaluator. After the evaluator changed, the same frozen
estimator was replayed on the current stack. Relative to that replay, the final
lawful estimator improved the adjusted score by 3.115%, yet the growing
board moved it to #40 by the end of the phase. The rank is historical, the
public board is provisional, and the fresh private rerun is what matters for
prizes.

What may be more useful to other participants is what happened between those
two snapshots. I ran 56 post-frontier capacity investigations. None yielded
a target-free raw-accuracy successor that survived the frozen gates. Exact
prediction-preserving compiler rewrites did survive, reducing metered compute
while leaving the estimator’s raw predictions unchanged.

This is the executable companion to my Phase 1 method write-up:
an evidence-indexed atlas of the whole search, including the branches that
failed. It is a Community Contribution research resource, not a second
Algorithmic Contribution entry and not a request to designate another
submission.

Across Phase 1 I materialized 203 numbered investigations spanning sampling,
deterministic cubature, moment and cumulant closure, covariance transport,
learned residual states, adaptive carrier selection, exact compiler rewrites,
and preregistered falsification. Eight labels were intentionally left unused;
the attached atlas does not pretend that a queued name was an experiment.

The attached bundle contains an executed notebook, a static HTML rendering, the
complete public-safe JSON branch table, two figures, dependencies, and explicit
reuse licenses. It contains no challenge targets, per-network identities,
private seeds, fitted public-row lookup, estimator source, or accounting-bypass
detail.

The hierarchy

A. Sampling and deterministic integration

I tested radial and antithetic Monte Carlo, shifted lattices and RQMC, complete
MUB rules, quantized Haar/Sobol sphere rules, multiple rotations, whitening,
projective and tangent controls, codebook selection, and point-count allocation.
Richer designs often reduced raw integration error, but the cheap target-free
selectors and allocation rules did not transfer strongly enough after compute.
One especially useful reversal was a public/development point-count improvement
that regressed on a 500-network external holdout (ratio 1.0934).

B. Scalar moments, Gaussian closure, cumulants, and mixtures

I tested diagonal and full-covariance Gaussian propagation, low-order cumulants,
factorized and projected K3 states, rank-one/rank-r covariance, scalar messages,
Gaussian mixtures, tensor trains, Hermite covariance, Edgeworth-style terminal
corrections, learned moment states, and Borel-Pade stabilization.

The recurring failure was not a lack of one more scalar calibration. Signed
cross-neuron dependence matters at depth 32, while small sufficient states
either lost it, became unstable, or exceeded the compute budget. Exact
Gaussian/ReLU covariance transport was mathematically reproducible, but the
dense state lost after compute; the tested Borel-Pade ladder had mean drift
9.765e-6 against a 1.650e-6 bound and only 0.297-0.398 stable fraction.

C. Complete Kerdock cubature and structural compaction

The one family that reached my official frontier was complete real Kerdock/MUB
angular cubature combined with exact Gaussian-radius conditioning, signed
FWHT evaluation, conservative analytic/pilot inactivity tests, analytic fill
for omitted inputs, and a dead/on/kink structural suffix.

The important boundary is that structural quality and metered execution became
separate problems. A sequence of exact rewrites preserved the same raw MSE
near 2.426e-7 while lowering mean effective compute from 166.897B to
161.368B and improving adjusted score from about 1.485e-7 to 1.439e-7.
That is a real score gain, but it is compute-driven rather than a new raw-
accuracy estimator.

D. Post-frontier capacity search

The attached table records 56 post-frontier rows. Representative results:

  • a target-informed coordinate-sign oracle showed large capacity, but the
    target-free proxy scored an adjusted ratio 2.0483 with only 2/8 wins;
  • six signed suffix features produced an optimistic adjusted ratio 0.9842,
    but later deployable descendants did not survive;
  • two local nonlinear heads had attractive train/oracle behavior but regressed
    untouched validation by 3.60-4.29x;
  • a full-Haar three-feature Stein control reduced train variance to 0.8461
    but regressed frozen validation to 1.1287, with 0/4 network wins;
  • a weighted 84-basis subframe passed a prediction-drift screen, then regressed
    frozen truth by 1.9761x, with 0/4 wins;
  • a 72-basis subframe regressed to 1.7714, with bootstrap q95 3.6657;
  • a learned all-depth K4 closure improved a diagonal baseline to 0.8094, but
    the hybrid against the strong deterministic carrier regressed to 1.0122;
  • a locally linear coreset had a zero-error compute lower bound 0.65625,
    already above the strategic 0.60589 capacity target; and
  • one-trimmed plane aggregation had pooled ratio 0.99798, but split 2/4
    networks and failed every frozen promotion rule.

Five conclusions I would reuse

  1. Oracle capacity and legal selection are different problems. Several
    truth-informed selectors exposed headroom; fixed target-free rankers failed
    to identify it across networks.
  2. A scalar terminal corrector is not the missing joint state. Energy,
    gate-count, alpha-bin, and small local-head families repeatedly failed
    independent transfer.
  3. Exact full covariance is an informative control, not automatically an
    efficient estimator.
    Mathematical fidelity can lose once the state cost is
    counted.
  4. Compiler improvements should be labeled honestly. They can improve the
    adjusted score while leaving raw prediction quality unchanged.
  5. Freeze before truth. Several persuasive development or public-mini wins
    reversed on network-disjoint reserves. Keeping those reversals in the
    record is more useful than a survivor-only story.

How to read the evidence

The notebook separates four evidence types:

  1. official grader readback;
  2. frozen network-disjoint synthetic holdout;
  3. source, compute, memory, or runtime gate; and
  4. impossible-oracle capacity screen.

They are not interchangeable. A public score is provisional until the fresh
private rerun. An impossible oracle can justify looking for a statistic but
cannot authorize a submission. A killed implementation narrows a branch; it
does not prove that the whole mathematical family is impossible.

Reproducibility and scope

The notebook executes top-to-bottom using only the adjacent curated JSON file,
pandas, and matplotlib. The public data table covers every grouped early
branch and every materialized post-frontier capacity row, while keeping the
private campaign’s operational material out of the release.

I used ChatGPT/Codex and Claude extensively for literature search, derivations,
implementation discussion, experiment design, drafting, and review. Numerical
claims were reconciled against frozen receipts or official grader readbacks.
The notebook code is MIT licensed; the narrative and curated data are CC BY 4.0.

I would especially value discussion of two open questions: what compact
target-free statistic can transport signed cross-neuron dependence through 32
ReLU layers, and what principled rule can select the real orientation/subframe
oracle capacity without observing targets?

Figures and executable bundle



Download the executable research-atlas bundle (215.1 KB)

Bundle SHA-256: 4a444787a008484967d6a25d98a67412ef6c0cbe188a50340be6cbe34d1ac72a

1 Like

Great write-up! Same happened to me with the early ranking and I used the Kerdock as well, I was feeling really good about my work up until then. I started throwing everything at it trying to go back up the leaderboard (even though it really doesn’t matter in the grand scheme of things).

I consolidated the methods and think I captured them correctly. I really like James’s method, if we were doing phase one again I’d be curious how well that and a few others presented could turn out, I believe this roughly summarizes them so far (corrections welcome).

Thanks — the table is useful, and your public raw/adjusted pair makes the compute–accuracy tradeoff unusually clear.

One wording correction for my row: I would label it “complete Kerdock/MUB 5-design + exact radial conditioning + analytic/pilot compaction + prediction-preserving compiler rewrites.” “Dead-compaction” by itself can read as though I am claiming the SOX dead/on/kink method; I explicitly am not. My narrower contribution was the metered route around the Kerdock carrier: signed-FWHT first-layer evaluation, conservative intersection of analytic and reused-pilot inactivity tests, analytic fill for omitted inputs, deterministic compacted continuation, and exact source rewrites.

Also, the final improvement was compute-only. Against the frozen estimator replayed on the same WhestBench 0.14.0 / FlopScope 0.10.0 stack, raw MSE stayed exactly 2.4258079704964076e-7; adjusted score moved from 1.4854216318155364e-7 to 1.43914441754321e-7 (3.115% lower).

Your fringe-planner row is the part I would most like to understand technically: public raw MSE 1.920e-7 is materially below mine. Which component produced most of that raw gain—the fringe allocation, the antipodal identity, or their interaction?

1 Like

Sorry for the late reply, but here 's my best answer: I think most of the raw accuracy came from the overall Kerdock/MUB cubature and pilot routing. The antipodal identity and fringe planner mainly reduced compute, and I didn’t isolate their individual contributions cleanly.

My two final entries were also different: #326948 was the pure Kerdock/MUB cubature version I chose as the public/write-up entry, while #327292 added a tail-four residual correction and was the version we originally suspected might transfer better to fresh networks. So the private comparison should be interesting. My best guess is the 498 will do better when the private grading goes through. I was pushing the moddle too hard and it started to make bad decisions and suggestions (I had /goal doing the heavy lifting and it was suggesting if I remove constraints it could get a better score, I reduced some and wasn’t entirely happy with the direction it went and was going in the simulations) make bad decisions and eventhough the 292 ended up with the lower overall score, i think it’s the inferior model of the two.