Phase 1 SOLUTION, ideas for Phase 2 and the honest public frontier (all methods combined and explained)

[thylinao]

I’ve spent the last 2 weeks gathering all topics and discussion from the forums and combined with my own 1.5 months long research I want to publish this page dedicated to explaining everything that was used in a beginner-friendly manner, suitable for someone who wants to understand it from 0 to where the current public knowledge frontier is. I’ve created animations, interactive figures, code samples and more for it to be engaging and accessible. On top of that many interesting levers emerged from posts and my own notes that I wanted to present them to everyone to test in Phase 2. All can be found below.

If I missed or misrepresented your method please let me know and I will edit it. All credits are given to the authors (if you want me to change the name I will happily do so).

Estimating a random network without running it

Many thanks to everybody!

[Cipo]

Thanks a lot for the report! I skimmed it and have one quick question: can I ask again how are you using the two observations of the section “What actually happens to the correlations” for your results? Or are they just observations?

Because my understanding was that the high correlations at deeper and deeper layers were pointing to a common mean vector: h_L(x)=\\mu_L+r_L(x) , a symptom of ReLU stacking funneling activations into some \\mu_L (indeed this is valid not only for opposite inputs). And the variance would live in r_L (this is what the estimator would try to capture well).

The second observation is: having rank of around 2, does not mean your error, measured from the variance, lives in a low dimensional space. If you want to get something competitive with sampling you would need many modes, say that capture 99.99% or 99.999% of the variance, in the classical sense.

Example of plot below to show that by just considering 99%:

Perhaps you used a different rank definition?

These were some of my thoughts, thinking about how to exploit these observations (and this would mean centered activation might be a better way to probe the structure of activations). However, I am curious about your thoughts!

I am happy to clear any of my misunderstandings/errors, so happy for clarifications.

[thylinao]

Thanks for reading it properly, and for the plot. You are right on both counts. I have fixed the page.

Are you using those two observations?

No, with one exception. They are there to explain why the closure fails, not to produce a number. The estimator is a scrambled Sobol sampler with no closure, no correction and no learned component in the scored path. Nothing in it looks at the correlations or the rank.

What actually drives the design is the oracle table just above that section. Give the estimator perfect marginals at every layer and leave only the correlations approximate, and it still cannot get below 7.07e-07. That is what says “spend the budget on samples”.

The one exception: the arc-cosine map is what killed antithetic sampling for me. Pairing x with -x is capped at 1.10x, and on a scrambled net it measured 0.93x, so it actually hurts.

The common mean vector

You are right.

Write h = mu + r. For a symmetric input distribution:

C = F + (1 - F) * rho

C is the antipodal cosine. F is the mean’s share of the activation energy. rho is the correlation between the two residuals.

Measured on ten fresh networks (256 wide, 32 deep, 131,072 samples each): F = 0.946, rho is about -0.08. So C and F agree to within 0.006.

In other words, “the correlations saturate near one” and “the activations are dominated by a common mean” are the same fact said twice. Yours is the better way to say it.

That also explains the antithetic result. The only thing antithetic sampling can harvest is the small (1 - F) * rho term, and it is tiny because almost everything the two vectors share is the mean.

One caveat on your framing. Here mu is not a nuisance, it is the answer, since the target is E[h_32]. The sampler’s error is trace(Cov_32) / (256 N). So r is the noise to average away, not the thing to model.

The rank definition

The 2.2 is a participation ratio, and your curve reproduces it:

	mine	yours
participation ratio, first post-ReLU layer	165	~165
participation ratio, scored layer	2.4	~2
99% count, first post-ReLU layer	246	~245
99% count, scored layer	56	~60

But the page called it “effective rank”, which is a different statistic. On the same spectra:

statistic	value
participation ratio (this is the 2.2)	2.4
effective rank, Roy and Vetterli, on eigenvalues	5.9
the same, on singular values	64
stable rank	1.6

So a perfectly defensible “effective rank” lands at 64, which is essentially your number. Most of the apparent disagreement was naming.

Your criticism of the sentence stands anyway. A participation ratio is dominated by the top eigenvalue and says little about how far the spectrum reaches. You can build spectra with a participation ratio of 2.4 whose 99% count is anything from single digits to over 200. Reaching 99% here takes 56 directions out of 256, and 99.99% takes 155. “Living on a two-dimensional object” is gone, with those counts in its place.

Needing many modes

Agreed, with one refinement. That ladder prices an oracle nobody can buy.

Capturing a mode means knowing E[v . h_32] along that eigendirection. Those are k of the 256 numbers the competition is asking you to produce. Knowing Cov_32 exactly buys nothing on its own, because the sampler’s error is just the sample mean of a mean-zero variable.

So the modes have to come from a surrogate whose mean you know in closed form. trim_qewas (18182) measured that whole class saturating at 0.47 of the variance, however many directions you add.

Centered activations

Agreed. Uncentered, the participation ratio is 1.04, which is near rank one only because the mean dominates, so it tells you nothing. Batch whitening is top of my Phase 2 list at a reported 2.06x to 2.34x.

What changed on the page

1. The rank sentence, as above.
2. The 0.982 cosine. It was compared against a theoretical value for a different quantity: 0.982 averages per-pair cosines, while the map predicts a ratio of expectations. That measures 0.941 against 0.973 from the map, so the map overpredicts the collapse rather than under.

Thanks for taking the time and the feedback!

[Cipo]

Thanks for your reply and checking stuff out.

Note, I never said \mu_L is not important or not useful. Just that in principle we care about residuals more for our current goals. I think.

One last thing:

@thylinao

What actually drives the design is the oracle table just above that section. Give the estimator perfect marginals at every layer and leave only the correlations approximate, and it still cannot get below 7.07e-07. That is what says “spend the budget on samples”.

Which marginals are you talking about?

I think others as well me used keenan’s dataset to check oracle limits.

For example:

Using pre-activation cumulants and going through the ReLU (ignore red dashed line). In principle, there is a lot of info to be gained by going beyond Gaussian.

If you do marginals I guess you are doing post-ReLU? Because marginals pre-ReLU would not work well, as a non-linear function will mix modes, and you need full objects.

[thylinao]

Fair point on μ_L and agreed that the residual is what matters for the current goal.

Which marginals: post-ReLU as you mentioned. The row says “true per-layer marginal mean and standard deviation after every ReLU, keeping the closure’s own correlations.” So at every layer, the estimator gets the exact post-ReLU mean and SD for each neuron. The only thing still being approximated is the cross-neuron dependence. Pre-ReLU marginals wouldn’t be a clean oracle for exactly the reason you mentioned.

Your plot and my table are two different oracles, and they agree where they overlap. Yours is a readout experiment: exact moments at a layer, then pass them through the ReLU. Mine has a readout row too, and the numbers line up:

	Yours	Mine
Gaussian closure, exact μ and σ	~1e-6	9.33e-07
+ third and fourth cumulant Edgeworth	~2e-8	1.5e-08 to 2.2e-08

So I agree there is real information beyond Gaussian in the readout. It’s worth roughly 50×, and it’s also i think the least controversial number in my table.

The catch is what happens when you also have to propagate. My 7.07e-07 row already uses a true-cumulant Edgeworth readout on top of exact post-ReLU marginals at every layer. That’s essentially your green curve applied at every layer, and the chain still bottoms out at 7.07e-07. The fact that the readout is nearly solved is precisely why I interpret the remaining error as dependence rather than marginal shape.

That also gives us a way to measure your “you need the full objects” point. Exact post-ReLU marginals at layer l don’t determine the pre-activation distribution at layer l+1, because that depends on the joint distribution. If you hand over every marginal for free and leave only the correlations approximate, you get 2.11e-06. Amalgonim’s shuffle oracle gets 9.2643e-07 from the other direction, which is in the same ballpark.

On the instrument: I didn’t use Keenan’s dataset. Mine was 500 self-generated networks, with the 2.00e-06 result replicated on a separate 150-network instrument. That i think is a main difference from what you and the others ran, and probably the most likely source of any disagreement in the absolute level.

Happy to rerun the two propagation rows on Keenan’s set so we’re comparing the same thing directly, if that’s useful. Thank you!