This post describes submission 319341 with adjusted score 1.551×10−7 and raw final-layer MSE 2.18e−7 as of Mon, July 27th at 6:30 pm ET. Please find the writeup attached, including a discussion of possible future directions and two ablation studies varying the activation function and network architecture. We highlight that our method does not easily transfer with architectural changes to the network. We would love to see people build on our solutions.
Summary: Our method is a hybrid estimator that uses structural observations about the network to improve upon pure Monte Carlo sampling. First, we use moment propagation to classify neurons as almost certainly inactive (dead), almost certainly active (on), or uncertain (kink). Neurons on the boundary between regions are reevaluated with a set of pilot samples and reclassified as appropriate. The three neuron classifications then drive subsequent sampling and implementation decisions. The kink and on neurons are evaluated using antithetic Sobol samples through layer 30. In the final two layers, the on neurons are treated linearly rather than sampling the ReLU activation function, while the kink neurons remain fully sampled. Even after removing dead neurons, there is significant sparsity in the sampled activations, which we leverage by sorting columns by firing rate and grouping rows by the number of active columns so that matrix multiplication can be performed more efficiently.
Code: We will publish our repo after the competition deadline.
AI use disclosure: Algorithm choices were data-driven based on human-led exploration, but the ideas were implemented in code by LLMs (Claude). An initial draft of the writeup was produced by LLMs (Claude, ChatGPT) and substantially revised by the authors.
main.pdf (6.1 MB)
*Minor post edits for grammar and formatting