Thanks a lot for the report! I skimmed it and have one quick question: can I ask again how are you using the two observations of the section “What actually happens to the correlations” for your results? Or are they just observations?
Because my understanding was that the high correlations at deeper and deeper layers were pointing to a common mean vector: h_L(x)=\\mu_L+r_L(x) , a symptom of ReLU stacking funneling activations into some \\mu_L (indeed this is valid not only for opposite inputs). And the variance would live in r_L (this is what the estimator would try to capture well).
The second observation is: having rank of around 2, does not mean your error, measured from the variance, lives in a low dimensional space. If you want to get something competitive with sampling you would need many modes, say that capture 99.99% or 99.999% of the variance, in the classical sense.
Example of plot below to show that by just considering 99%:
Perhaps you used a different rank definition?
These were some of my thoughts, thinking about how to exploit these observations (and this would mean centered activation might be a better way to probe the structure of activations). However, I am curious about your thoughts!
I am happy to clear any of my misunderstandings/errors, so happy for clarifications.