Hi guys,
given phase 1 just ended, I was thinking we could open some discussion about our approaches, lessons learnt, and anything else we found interesting.
I’ll start by sharing some thoughts.
I had little hope of immediately improving the mechanistic algorithm at L32. I tried a few approaches. Then, I directly went for a hybrid version (I think several of us did).
The general idea was to build such a hybrid version, then reduce as much as variance as possible.
-
First, we can use a deterministic quadrature rule. I tried several, and like many of you I ended up with a Kerdock/MUB bank, paired with simple basic techniques to reduce variance (like antithetic variables). It turns out, that for our kind of problem Kerdock works pretty well, as it gives a nice angular coverage over the integration space.
-
The second main idea, was feeding in some control variate. I come from an astrophysics field where control variates are pretty popular. In my case I did something pretty basic: if you look at different layers, say layer 10 and layer 32, you should expect to be correlated: say have a fluctuation \delta x_L, then another layer L'>L should respond to that fluctuation, say \delta x_{L'}, for x_L being the post-activation at layer L. Hence, you can then make a first approximation, and just write \delta x_{L'}=R_{LL'}\delta x_{L} (no Einstein summation here). This was my control variate. At that point the difficult parts were: get some nice analytic approximation, e.g. given by standard K=3 analytic methods with some fitting, and make sure your variate removes the error generated by your quadrature rule. It turns out this worked pretty great. In my case, I am able to reduce MSE at final layer by 50%.
These two ingredients were pretty much the base of my submission, see plot below (A716 is a particular analytical curve with some fitted coeffs).
Hope to share soon a technical and more rigorous pdf.
Some extra:
- like many of you, I used common tricks to reduce number of FLOPs of matrix multiplication. Note, if you read Wu’s ARC paper, they explicitly avoid this when comparing mechanistic methods with sampling. I ended spending pretty much my last day on this, as most of my time was not spent in making compute more efficient, beyond what the AI proposed, but in finding better control variates, as I think the space to improve there is huge. Without these matrix techniques, my effective compute (effectively) doubles!
- I also used a mix of other tricks, but at this point this was more like engineering and less like understanding the fundamentals of the problem
- I am not sure if QMC is really sampling, but it is not pure mechanistic (perhaps some people have a different definition). I believe there is a lot to be done in this space (unless top people already figured out something here). One idea is to find a mechanistic way to select your quadrature points based on your theoretical understanding. There should be a few ‘directions’ along which you really care for getting amazing results.
Thoughts for the future:
- going back to the last point, given your deterministic quadrature rule, you could manage to use very little compute to select the right ‘frames’ over which to get an excellent estimate of the (quenched) mean. However, I did not manage to find any deterministic way of getting say the best Kerdock points tailored to the net (I tried a few methods, but was not better than randomly choosing points)
- I found out pretty quickly that the concept of low-rank was important, and in the beginning of phase 1 I ended up with some basic algos that were quite useful (but did not use here)
- even if we do not get a better analytical method at L32, I believe even improving over current state of the art around L16 or L20 (for n>200) would be a big step, as CV methods would get much better
I am especially curious about your thoughts for the future. I think this kind of problems are quite fun, and given they are currently neglected there is a lot of space to do something fun and relevant!