[Phase 1 write-up] Fixed grid of directions instead of random sampling (submission #320802) and five things that did not work

skye_nygaard · August 10, 2026, 5:45pm

Phase1_Algorithmic_Contribution_320802.pdf (24.4 KB)

I am #47 right now, so this is not a winning method. It is a post about a mechanism, plus a pile of things that did not work, which I think are the more useful part.

I have tried to write this so you can follow it without knowing anything about cubature or coding theory. Terms are explained the first time they appear.

You are handed the weights of a neural network and asked: on average, how strongly does each neuron fire when you feed the network random inputs?

The obvious answer is to try it. Feed in thousands of random inputs, run the network, average what you see. That is Monte Carlo sampling, and it works, but it converges slowly — to halve your error you need four times as many samples. The competition gives you a compute budget about 15,000x smaller than the reference answer was computed with, so simply sampling harder is not available.

What I did

Replace random sampling with a fixed, carefully chosen set of directions.

The inputs are drawn from a standard Gaussian. The network has no bias terms, and ReLU has the property that scaling the input scales the output by the same factor. Put those together and only the direction of the input matters, not its length. The question “what is the average activation?” becomes “what is the average over all directions?”. This is equivalent to an average over the surface of a sphere in 256 dimensions.

Averaging over a sphere is a problem with a long history. You do not need random points. You can use a fixed list of directions chosen so that their average is very close to the true average. The same idea as Simpson’s rule for ordinary integrals, but on a sphere. Such a list is called a cubature rule.

I used a list of 66,048 directions with a property called being a spherical 5-design: averaging any polynomial of degree 5 or less over these directions gives exactly the right answer, not an approximation. The construction comes from the Kerdock code, an object from coding theory, together with a related family of bases. Concretely: 128 Kerdock bases plus the ordinary coordinate basis, 256 directions each, and every direction paired with the direction opposite to it.

That pairing is free accuracy. Including every direction alongside its opposite automatically cancels all the odd-degree error terms, so a 5-design costs no more than a 4-design would.

The list is chosen once, frozen into a file shipped with the submission, and reused unchanged for every network. It never looks at the weights.

Applying 66,048 directions to a 256x256 weight matrix should mean a large matrix multiply. It does not, because Kerdock bases are built from a structured pattern that can be applied by a Walsh-Hadamard transform. This is a fast transform in the same family as the FFT, which does the job in eight cheap passes instead of one expensive multiplication. Accuracy comes from the geometry of the point set and speed comes from the algebra underneath it.

I held everything statistical fixed — same 66,048 directions, same radius, same rows returned, same 100 development networks — and changed only the arithmetic used to push the directions through the network’s layers:

arm	raw final MSE	effective compute	budget	adjusted
ordinary matrix multiplication	2.2826e-7	2.689e11	98.86%	2.2565646e-7
Strassen–Winograd, depth 5	2.2819e-7	1.748e11	64.27%	1.4641716e-7

Raw error is unchanged to three significant figures. The score improves 1.5412x, entirely because the second technique is cheaper.

Strassen–Winograd is a trick for multiplying matrices with fewer multiplications than the schoolbook method. It gives the exact same answer, not an approximation. Applied five levels deep it charges 7^5 = 16,807 multiplications where ordinary blocking charges 32^3 = 32,768, a 1.9497x reduction.

Why I stopped looking for a better set of directions

I asked the question: how much room is there in this approach at all? If I had spent the whole competition hunting for a cleverer set of directions, how much better could I possibly have done?

The answer turned out to be: almost none. For this specific problem — 256 dimensions, 32 layers, a budget of 66,048 directions — I had AI produce a code proof for two bounds:

Against every possible choice of directions with positive weights, the Kerdock set is at most 0.0233242% worse than the best one that exists.
Even allowing negative weights, no fixed rule can beat Kerdock by more than 6.2940%.

So the set of directions was essentially already optimal, and moving points around or reweighting them was a dead end. That is why the remaining budget went into the arithmetic instead. The ceiling result is the reason the Strassen–Winograd work happened at all.

Two limits on this. First, these bounds apply to fixed rules — a set of directions chosen in advance, the same for every network. They say nothing about methods that look at the specific network they were given, adapt as they go, or combine results non-linearly. That is very likely where the leaders here are working, and nothing I proved constrains them. Second, the proof is computer-assisted and replayable but not independently verified.

Some of the claims have more nuance than stated here. The nuance is in the repo

Five things that did not work

Tracking the spread of activations, not just the average, from one shared reference point. As signals pass through layers they spread out into something like a mixture of several clusters. I tried describing all of them relative to one common centre. Measuring from the pooled centre did shrink the spread terms a lot (0.574 → 0.357 at layer 29), and error got worse anyway (4.00e-3 → 5.41e-3). The reason is structural: the error is dominated by how far apart the cluster centers are, and the method only improves by separating those centers further. It necessarily degrades as the representation gets better.
Compressing the layer-to-layer transformation to a low-rank approximation. Keeping only the r most important components gave relative errors of 2.16e-1, 5.44e-2, 6.73e-3 and 7.86e-4 at r = 4, 16, 64 and 128. The accuracy I needed was about 1.5e-3, so only r=128 was good enough — but the budget only afforded about r=4. And at r=128 the “compressed” version costs the same as the uncompressed one.
Using fewer directions. Dropping from 129 bases to 96, 64 and 32 made the score worse by 1.3743x, 1.4096x and 1.5579x respectively. The reason is that error grows roughly as k^-1.21 to k^-1.24 in the number of bases k, and any exponent above 1 means error rises faster than the compute saving falls. The full set always wins. A reduced set is only worth it as a cheap base for some correction on top and at 96 bases that correction would have needed to deliver a 1.2841x accuracy gain to break even, against a projection that said 1.0670x would do. Nothing I had came close to either number.
Correcting the final layer against an anchor point. The value I propagate sits about 0.65% off, and about 0.45% of that is eaten by the cost of computing a correction. Roughly 0.3% of usable headroom is not enough to build anything on.
Learned correction models and hand-designed features of the weights. Several variants. None of them paid for their own compute.

What is public

github.com/SkyeNygaard/arc-whitebox — MIT.
arc_whitebox/submissions/production_baseline_320802/ — the estimator behind #320802: the estimator, the fast matrix-multiply kernel, the frozen direction set, and the submitted archive (sha256 77be0e88…, passes whest validate-package), plus both development result bundles.
whestbench/phase1_320802.json — a machine-readable record tying the submission id to its graded numbers, file hashes, method, cost, experiments, negative results and limitations.
theory/ — the paper and proof archive behind the ceiling result above.

LLM use

Heavy.

I think this competition was super helpful for me to get moving. A clear goal for LLM to optimize. ChatGPT gives you unlimited token usage within the normal model and $20 a month tier gives you access to that and the “library” where models can share files. I kept a research ledger shared across and could run roughly ~10 agents at once. The LLMs are really bad at backing out of research and trying new paths. Want to optimize, and want to cheat to achieve object (find a way for compute to not be measured). You need to babysit and be very strong about changing direction. I suspect they will be better at next release. I also used codex set to ultra, which was much more robust, better to spawn sub-agents with only the relevant context and not full context every time.

Opus 5 is pretty bad at math, Fable 5 is much better but too expensive. Recommend people try using chatgpt through web interface, it will run things in sandbox, can really use a lot of tokens.

I posted all the research directions LLMs tried in a ledger in my repo. Anyone can have their own LLM look at it to see directions that are open, but I think LLMs tend to get stuck in a certain path and not think broadly. I think that’s the best value humans add for LLM math research.