I didn’t end up getting anything working before the submission deadline, but I’ve been exploring a whole range of avenues towards alternatives to the cumulant propagation algorithm. So this write-up is another set of notes collecting some of my current thinking and ideas I played with.
notes_on_the_arc_challenge_problem_part_2.pdf (651.5 KB)
(100% human generated, cruft and all)
The top-scoring submissions appear to be quite homogeneous in effectively just tweaking ARC’s cumulant propagation algorithm. For anyone who was interested in new ideas beyond simply optimizing code for the same ole’ thing, you might find something interesting here.
In my previous set of notes I argued that structure in \vec{F}(\vec{x}) - that is, the Fourier harmonic content - is essentially isotropic when viewed as a black-box because the matrices W_\ell are drawn independently from an isotropic distribution. However, if we consider each layer function individually, the columns of the weight matrices are a set of special directions. This is shown explicitly in Section 5. I think this structure will shed light on the failure of the cumulant propagation scheme to scale with depth, and points the way towards doing better.
The cumulant propagation described in ARC’s paper is implicitly modeling the intermediate distribution of each layer 's output by a truncated Edgeworth/Gram-Charlier series but the choice of the standard component basis is misaligned with the important structure. When the series so defined is truncated to the very low order necessitated by finite available compute, it is effectively looking at only a finite set of marginal distributions in directions that are relatively insensitive to the important non-Gaussianity. I’m thinking that capturing this anisotropic non-Gaussian structure more efficiently is how you improve the efficiency of a closure propagation and hence the final estimate.