Phase 1 mechanistic-estimation research atlas

A public-safe, reproducible map of 203 materialized numbered investigations into estimating activation means of width-256, depth-32 random ReLU MLPs. This is a Community Contribution research resource, not a second Algorithmic Contribution entry.

tl;dr

1. Complete Kerdock angular cubature plus conservative structural compaction was the only mechanism family that reached our official frontier.
2. Exact compiler rewrites moved the adjusted score from about 1.485e-7 to 1.439e-7 while preserving raw MSE near 2.426e-7.
3. Scalar moments, small terminal heads, low-rank summaries, learned closures, and rich target-free selectors repeatedly failed independent transfer gates.
4. Several impossible oracles showed real information capacity, but fixed legal proxies could not reliably select it.
5. The strongest practical lesson is methodological: distinguish source gates, frozen network-disjoint holdouts, impossible-oracle screens, and official grader readbacks instead of treating them as one leaderboard.

Context & Methods

The source campaign used numbered branches, preregistered promote/kill rules, network-disjoint synthetic holdouts, and current-stack package replay. This notebook contains only a curated public extract.

Key Assumptions

• Lower adjusted score and lower raw MSE are better.
• Effective compute is reported in billions of FLOP-equivalent operations.
• Official public scores are provisional until the fresh private rerun.
• A failed implementation narrows a branch; it does not prove an entire research family impossible.
• No private seeds, per-network identities, public-target fitting, or accounting bypass details are included.

from pathlib import Path
import json

import matplotlib.pyplot as plt
import pandas as pd

from IPython.display import Markdown, display

DATA_PATH = Path('phase1_research_atlas_data.json')
with DATA_PATH.open() as handle:
    atlas = json.load(handle)

print(atlas['title'])
print('Materialized numbered investigations:', atlas['materialized_numbered_investigations'])
print('Intentionally unused labels:', ', '.join(atlas['intentionally_unused_labels']))

Phase 1 mechanistic-estimation research atlas
Materialized numbered investigations: 203
Intentionally unused labels: e187, e188, e189, e191, e192, e193, e194, e195

Data

The data file has three grains: official frontier points, grouped pre-frontier mechanism branches, and one row per post-frontier capacity investigation.

frontier = pd.DataFrame(atlas['official_frontier'])
family_rows = pd.DataFrame([
    branch
    for family in atlas['mechanism_families']
    for branch in family['branches']
])
post_e141 = pd.DataFrame(atlas['post_e141_capacity_search'])

print('Official frontier rows:', len(frontier))
print('Grouped early mechanism rows:', len(family_rows))
print('Post-e141 capacity rows:', len(post_e141))
frontier[['experiment', 'adjusted_score', 'raw_mse', 'effective_compute_billions', 'meaning']]

Official frontier rows: 8
Grouped early mechanism rows: 20
Post-e141 capacity rows: 56

	experiment	adjusted_score	raw_mse	effective_compute_billions	meaning
0	e110	2.270270e-07	1.955730e-06	32.289	low-compute compaction frontier
1	e136	1.737380e-07	2.417830e-07	195.848	pre-e141 Kerdock frontier
2	e141	1.481100e-07	2.425800e-07	166.289	score and compute incumbent
3	e141-final	1.485420e-07	2.425810e-07	166.897	released-stack calibration
4	e199	1.479330e-07	2.425810e-07	166.231	exact compiler incumbent
5	e201	1.475540e-07	2.425810e-07	165.813	exact compiler incumbent
6	e204	1.453590e-07	2.425810e-07	163.347	exact compiler incumbent
7	e211	1.439140e-07	2.425810e-07	161.368	exact compiler incumbent

Results

1. Mechanism hierarchy

for family in atlas['mechanism_families']:
    print(f"{family['family_id']}. {family['name']}")
    print('   Conclusion:', family['conclusion'])
    for branch in family['branches']:
        print(f"   - {branch['experiments']}: {branch['branch']} -> {branch['decision']}")
    print()

A. Sampling and deterministic integration
   Conclusion: Richer designs often reduced raw integration error, but tested target-free selectors, whitening rules, codebooks, and point-count tuning did not transfer strongly enough after compute.
   - e002-e008: radial, whitened, antithetic Monte Carlo and simple blends -> control family; extra independent passes do not win after compute
   - e014, e021, e046: shifted rank-1 lattice / direct RQMC / SphereMLP -> closed in tested forms
   - e019-e020: complete 17-real-MUB and sampling blends -> raw improves, adjusted regresses
   - e022-e026: quantized Haar/Sobol sphere and quantizer variants -> contribution artifact retained; not leaderboard incumbent
   - e030, e056, e061-e062, e066, e068-e075: multiple Haar rotations, codebook selectors, harmonic/tangent/pair controls -> only e069 is a verified component
   - e047, e049-e055: point-count ladder and adaptive allocation -> pure point-count tuning closed
   - e058-e059: whitening and projective controls -> closed
   - e071-e072: near-orthogonal blocks and low-count tangent -> closed

B. Scalar moments, Gaussian closure, cumulants, and mixtures
   Conclusion: Signed cross-neuron dependence matters at depth 32. Scalar, diagonal, small low-rank, low-order Hermite, factorized K3, and compact learned states did not transport enough of it under the budget.
   - e001, e013, e015-e018: diagonal/full-covariance Gaussian and scalar rolled-state corrections -> useful anchors, not contenders
   - e009, e017, e024, e027-e029, e032-e033: exact/factored/projected K3 and tensor compression -> tested carriers closed
   - e012, e016, e031, e036-e043: learned moment closures, rank-one/rank-r covariance, scalar messages -> closed in tested sufficient states
   - e034, e041-e042: affine-hinge and one-message recurrences -> closed
   - e040, e045: Gaussian-mixture split/merge -> closed
   - e060, e067: tensor train and low-rank multifidelity -> closed
   - e079-e081: pairwise teacher, Hermite covariance, monomial NFN -> new state required
   - e086-e098: NTK/MMD, K3, Edgeworth, tangent, adaptive-oracle and learned terminal variants -> closed in exact tested forms

C. Complete Kerdock cubature and legal compute
   Conclusion: Complete Kerdock cancellation survived, but remote runtime and metered data movement became independent design constraints. Exact rewrites improved score without changing predictions.
   - e094, e096, e110: complete Kerdock carrier plus exact dead compaction -> established the carrier and compaction route
   - e117, e122-e132: Strassen/Winograd, basis changes, chunks, subsets, scalar mean -> runtime compatibility is an independent gate
   - e130, e134-e136: scalar mean and two/three-layer folding -> superseded by e141
   - e133-e141: dtype, fold, dead/cold/kink/on compaction -> incumbent

2. Official accuracy-compute frontier

fig, ax = plt.subplots(figsize=(9, 5.5))
ax.scatter(
    frontier['effective_compute_billions'],
    frontier['adjusted_score'] * 1e7,
    s=70,
    color='#2b6cb0',
)
for row in frontier[frontier['experiment'].isin(['e110', 'e136'])].itertuples():
    ax.annotate(
        row.experiment,
        (row.effective_compute_billions, row.adjusted_score * 1e7),
        xytext=(5, 5),
        textcoords='offset points',
        fontsize=9,
    )
compiler_cluster = frontier[~frontier['experiment'].isin(['e110', 'e136'])]
ax.annotate(
    'six frozen Kerdock revisions\nexpanded in the second figure',
    (compiler_cluster['effective_compute_billions'].mean(), compiler_cluster['adjusted_score'].mean() * 1e7),
    xytext=(118, 1.63),
    textcoords='data',
    arrowprops={'arrowstyle': '->', 'color': '#4a5568'},
    fontsize=9,
)
ax.set_xlabel('Mean effective compute (billions)')
ax.set_ylabel('Adjusted score (×1e-7, lower is better)')
ax.set_title('Official team frontier: accuracy and metered compute')
ax.grid(alpha=0.25)
fig.tight_layout()
fig.savefig('phase1_official_frontier.png', dpi=180, bbox_inches='tight')
plt.close(fig)
display(Markdown(
    '![Scatter plot of adjusted score against mean effective compute for eight official team frontier experiments.]'
    '(phase1_official_frontier.png)'
))

The low-compute point is intentionally retained as a control. The later staircase shows exact prediction-preserving compiler work: raw MSE stays fixed while metered compute and adjusted score fall.

3. Prediction-preserving compiler staircase

staircase_order = ['e141-final', 'e199', 'e201', 'e204', 'e211']
staircase = frontier.set_index('experiment').loc[staircase_order].reset_index()
staircase['compute_saved_vs_first_b'] = (
    staircase.loc[0, 'effective_compute_billions']
    - staircase['effective_compute_billions']
)
staircase['adjusted_improvement_pct'] = (
    1 - staircase['adjusted_score'] / staircase.loc[0, 'adjusted_score']
) * 100
staircase[['experiment', 'effective_compute_billions', 'compute_saved_vs_first_b', 'adjusted_improvement_pct', 'raw_mse']]

	experiment	effective_compute_billions	compute_saved_vs_first_b	adjusted_improvement_pct	raw_mse
0	e141-final	166.897	0.000	0.000000	2.425810e-07
1	e199	166.231	0.666	0.409985	2.425810e-07
2	e201	165.813	1.084	0.665132	2.425810e-07
3	e204	163.347	3.550	2.142828	2.425810e-07
4	e211	161.368	5.529	3.115617	2.425810e-07

fig, ax = plt.subplots(figsize=(9, 4.8))
ax.plot(
    staircase['experiment'],
    staircase['effective_compute_billions'],
    marker='o',
    linewidth=2,
    color='#2f855a',
)
for row in staircase.itertuples():
    ax.annotate(
        f"{row.effective_compute_billions:.2f}B",
        (row.experiment, row.effective_compute_billions),
        xytext=(0, 8),
        ha='center',
        textcoords='offset points',
        fontsize=9,
    )
ax.set_ylabel('Mean effective compute (billions)')
ax.set_xlabel('Frozen estimator revision')
ax.set_title('Exact rewrites saved compute without changing raw predictions')
ax.grid(axis='y', alpha=0.25)
fig.tight_layout()
fig.savefig('phase1_compiler_staircase.png', dpi=180, bbox_inches='tight')
plt.close(fig)
display(Markdown(
    '![Line plot showing mean effective compute falling across five exact prediction-preserving revisions.]'
    '(phase1_compiler_staircase.png)'
))

4. Post-frontier negative-result atlas

pd.set_option('display.max_colwidth', 110)
post_e141[['experiment', 'operator', 'decisive_result', 'status']]

	experiment	operator	decisive_result	status
0	e142	scalar conditional energy/product	depth-32 error about 2,200x the rank-5 raw ceiling	killed
1	e143	two independent half rotations	variance ratio 1.13891 due negative half covariance	killed
2	e144	exact batched cold microkernels	slower than analytic-dead route	killed
3	e145	on32 crossing correction	adjusted ratio 1.0096	killed
4	e146	target-dependent coordinate-sign oracle	adjusted ratio 0.5688	capacity only; not deployable
5	e147	target-free coordinate-sign proxy	truth adjusted 2.0483; 2/8 wins	killed
6	e148	fixed Walsh pair/gate fields	cross-replicate adjusted 0.99315	killed
7	e149	balanced within-basis row subsets	adjusted 1.5881	killed
8	e150	centered-SVD range carrier	best quality drift 1.29x triangle bound; best-compute drift 5.65x	pure SVD killed; e182 closed the suffix-aware correction
9	e151	compute audit	87.0% of FLOPs are in compact prefix layers 2-29	directs new work to the prefix
10	e152	q=1 exact-line preintegration	optimistic capacity ratios above 2,731x anchor	tested directions killed
11	e153-e154	six signed suffix features / two templates	refined adjusted 0.9842, q95 0.9966	killed
12	e155	transferred tangent on e141	oracle 1.1135; beta transfer 1.00327	killed
13	e156	uniform layer-29 JVP/beta	beta=1 ratio 3812.9; oracle 1.184	killed
14	e157	minimal late-kink linear head	deployable 1.0097; oracle 1.0038	killed
15	e158	shared nonlinear local heads	validation 3.60-4.29 despite oracle 0.593	killed
16	e159	alpha-tail ridge	validation adjusted 1.9233	killed
17	e160	terminal Gaussian gate-count control	first fresh MLP health p=3.54e-5 below 1e-4 floor	killed before truth
18	e161	six-bin alpha calibration	untouched validation raw 1.01482; one OOF fold regressed	killed
19	e162	official smoke-v2 compatibility	zero failures; adjusted 1.356e-5 on one N=1,024 row	compatibility only; target noise is over 50x the official frontier
20	e163	exact full-covariance Gaussian/ReLU transport	exact/Hermite-4 raw 1.00009, adjusted 1.24332; 3.43x effective compute	dense and block-sparse covariance branches killed
21	e164	e141-native mask/fold capacity oracle	impossible adjusted 0.94401; bootstrap q05 0.80146	killed before deployable port
22	e165	1/32 sketched-L2 coordinate-sign selector	dev raw 1.10421, adjusted 1.21872; no fresh seed used	killed before truth; frozen Walsh lookup was not deployable
23	Pro PF-ERC	recurrent pair/fiber residual state	width-32 synthetic capacity 0.68364; cheap weight proxy 0.99572	information-only signal; tested recurrent port failed
24	e166	empirical layer-29 covariance suffix	optimistic cross-fit 0.98053; fixed replacement 19.648; tau 0.64176	killed before deployable port
25	e167	public pair/fiber teacher-forced transition	reserve ratio 1.00200; bootstrap upper 1.00510; affine no-state 0.78844	killed before recurrent rollout
26	e168	exact-e141 sign bias/variance decomposition	variance-share bootstrap q05/q50/q95 0.77332/0.93054/1.08893	diagnostic: fixed antithetic coupling is the sole live extractor
27	e169	fixed relative-sign half carrier	official adjusted 1.73384e-7; e169/e141 1.17064; raw ratio 1.15912	official probe killed C1 family; incumbent unchanged
28	e170	full-Haar p1/p3/p5 Stein control	train variance 0.84613; validation 1.12874; q05 1.06937; 0/4 wins	killed before truth
29	e171	e167 affine no-state dependency audit	x1/x1a 99.7865% of gain; rn ratio 0.99619; proxy cost fits cap	static kill: no legal frozen final-mean operator
30	e172	rich eight-sign full-depth pilot selector	25.202B FLOPs; 35.139B-42.253B selector effective compute	frozen 35B design exceeded before labels or validation truth
31	e173	top-two joint-gate Gaussian suffix	candidate carrier-mean MSE 9706.06x e141 at 5.891B added FLOPs	algebraic kill before prediction freeze or synthetic truth
32	e174	distilled 84-basis Kerdock subframe	full-e141 drift 9.7415e-9; effective-compute ratio 0.72047 > 0.68	target-free compute kill; reserve truth remained unopened
33	e175	multi-sign cost/information frontier	best LOO adjusted 0.72353; every ranker failed positive rank transfer	tested rich-selector family killed without validation truth
34	e176	distilled 72-basis Kerdock subframe	fresh adjusted 1.77141; bootstrap q95 3.66568; 1/4 network wins	killed after frozen network-disjoint truth
35	e177	exact signed layer-31 ridge control	orientation-variance x compute 1.02831; q95 1.04921; 1/4 network wins	killed before truth; dependent-basis LCMV descendants closed
36	e178	final-three-layer equivariant learned state	reserve ratio 0.99924; q95 1.21923; 1/4 network wins; 1.129B FLOPs	capacity screen killed; no FlopScope estimator port or GPU run
37	e179	complete-basis K=16 late-kink soft LCMV	fresh adjusted 3.67180; q95 7.37381; 0/4 network wins	toy effect reproduced, then killed on two frozen truth replicates
38	e180	rank-4 observability retained law at L-1	deployment 0.99473, q95 1.00033; capacity 1.02382, q95 1.09436	killed at frozen L-1 gate; rank 2 and L-2 remained unopened
39	e181	gate-population tensor train	rank-2 ratio 0.99517, q95 1.05458; rank-8 carrier-identical	killed at width 16; no width-32/256 escalation
40	e182	suffix-aware low-rank checkpoint-31 map	legal adjusted 15.99120, raw 16.08722; compute 0.99569; 0/4 wins	killed on corrected frozen reserve; worker split was invalid
41	e183	four-phase within-basis Kerdock carrier	source-gate ratio 12.45517; 0/16 orientations, 0/4 nets	killed before legal prediction freeze or official submission
42	e184	rank-32 retained spectral law at L-1	independent PCA ratio 6.18121, q95 9.81510; observability 6.22998	killed on 4-network frozen reserve; 0/4 wins, no GPU or learned map
43	e185	depth-32 NNGP-MMD carrier deformation	exact e141 proxy 2.42891e-7; best tight-frame floor ratio 0.99927323	static kill: at most 0.0727% surrogate gain under stated constraints
44	e186	16-region empirical-Jacobian Fisher rotation	source-only adjusted 1.18486; 0/2 wins; 180B-184B effective	killed before truth; exact input-conjugacy health passed
45	e190	Borel-Pade resummation of K1/K2/K3 ladder	mean drift 9.765e-6 vs 1.650e-6 bound; stable fraction 0.297-0.398	current-stack source kill before truth or packaging
46	e196	activation-stratified local-linear coreset	zero-error compute lower bound 0.65625 vs 0.60589 capacity target	killed before truth, source port, packaging, or submission
47	e197	means-500k static weight-moment residual	reserve ratio 14.38095, bootstrap q95 15.77598, 0/4 network wins	frozen low-noise reserve kill; no GPU, source port, or submission
48	e198	all-depth K4 recurrent learned closure	final-mean ratio 0.80943 vs diagonal; e201 hybrid 1.01220, q95 1.02251	closed after frozen 256-network reserve and 64-network hybrid screen
49	e199	exact e141 pilot-input compaction	official 1.479332e-7 adjusted; raw exactly equal to e141-final	promoted current-stack incumbent; no sibling tuning
50	e200	64-path carrier-conditioned learned closure	reserve ratio 4.77095; q95 6.01539; 0/4 network and fold wins	frozen reserve kill; no port, GPU payload, package, or submission
51	e202	Kerdock plane/gate-moment capacity oracle	validation ratio 5.55812; q95 20.37721; 0/4 fixed block wins	hard reserve kill; no source port, package, GPU, or submission
52	e208	terminal omitted-dead variance smoothing	exact-carrier proxy ratio 1.00014384; 0/1 fresh network wins	target-free precheck kill; no truth, source port, package, or upload
53	e209	one-trimmed Kerdock plane mean	pooled truth ratio 0.99798124; replicate ratios 1.00151086 / 0.99504753	terminal 2/4-network kill; no source port, package, or upload
54	e210	stack-3/chunk-64 exact compiler schedule	bitwise parity; +2.348B FLOPs; +5.171B effective compute; 0/4 paired wins	terminal smallest-gate kill; no package or upload
55	e211	delete already-local identity gathers	bitwise parity; -1.658B mean FLOPs; 16/16 effective-compute wins	frozen package survivor; one official calibration authorized

The table deliberately mixes source kills, network-disjoint holdouts, and impossible-oracle capacity checks. The evidence type is part of the result; a promising oracle is not a deployable estimator.

5. Transferable findings

for index, item in enumerate(atlas['transferable_findings'], start=1):
    print(f"{index}. {item['finding']}")
    print('   Evidence:', item['evidence'])

1. Complete angular designs can beat simple sampling under a deep-network budget.
   Evidence: The complete Kerdock route reached an official raw MSE near 2.43e-7, then exact compute reductions lowered the adjusted score while preserving that raw MSE bit-for-bit.
2. The missing information is not well summarized by one scalar energy or a tiny terminal head.
   Evidence: Scalar conditional-product, gate-count, alpha-bin, local-head, and several learned closure families failed source gates or network-disjoint validation.
3. Oracle capacity and deployable selection are different problems.
   Evidence: Several target-informed orientation or subframe oracles showed large headroom, while fixed target-free rankers failed to transfer across networks.
4. Exact full covariance is mathematically useful but not automatically score-efficient.
   Evidence: The exact Gaussian/ReLU covariance formula was reproduced, yet its cost-adjusted candidate lost after accounting for the dense state.
5. Freeze-before-truth discipline prevented multiple attractive false positives.
   Evidence: Public-mini, development, or train-only wins repeatedly reversed on network-disjoint reserves; the atlas keeps those reversals visible.

Takeaways

• The successful Phase 1 path combined a strong deterministic carrier, exact radial structure, conservative analytic/pilot compaction, and metered execution.
• The large residual gap is unlikely to close through another scalar correction or small compiler tweak. A useful Phase 2 branch needs a new target-free joint-law statistic or a principled way to select the real oracle capacity.
• Negative results are most reusable when they state the tested sufficient state, the evidence type, the frozen decision rule, and the exact reason the branch was closed.

Limitations

This atlas summarizes one campaign rather than proving impossibility theorems. Read every kill as a scoped statement about the tested implementation and gate. See the bundled JSON for the complete public-safe branch table.

LLM disclosure and license

ChatGPT/Codex and Claude were used extensively for literature search, derivations, implementation discussion, experiment design, drafting, and review. Numerical claims in this notebook were copied from frozen campaign receipts or official grader readbacks and then independently reconciled against the source experiment map.

Notebook code: MIT. Narrative and curated data: CC BY 4.0.