Hello all,
Thank you for joining the ARC White-Box Estimation Challenge Townhall. Here’s a summary of what was covered:
Key takeaways:
- Team formation and merging is open until the end of Phase 1. You can keep submitting throughout Phase 2, and any submission during that window can be designated as your final entry. The designation locks at the actual Phase 2 deadline.
- Scoring is based on final-layer MSE only. Intermediate layer stats shown in the interface are for debugging, not scoring.
- To win a prize, you must release your code under an Open Source Initiative approved license. Declining forfeits the prize to the next eligible team. Community contribution prize submissions are released regardless.
- LLM use is fully permitted and encouraged, but must be disclosed transparently, especially if a write-up or code was largely LLM-generated.
- Offline training on external Monte Carlo data (e.g. published MLP moment datasets) is within the rules and untracked, but is seen as less likely to earn the algorithmic contribution prize than a more theoretical/mechanistic approach.
- The private rerun evaluates designated entries on a completely fresh set of MLPs. No public-split score carries over.
- The grading dashboard is expected to get updates: more decimal digits and a direct compute multiplier (C/B) display are planned.
You can watch the Town Hall recording here.
The townhall also discussed the following points in more detail:
Prizes and Judging
- Top prizes are based on MSE performance within the compute (flop) budget, with a modest discount if the full budget isn’t used.
- The algorithmic contribution prize is more discretionary. ARC will review roughly the top 10 submissions (possibly more), looking for meaningful mechanistic analysis that measurably improved the score, not just black-box sampling with minor enhancements. A PDF write-up is required (strongly recommended at minimum) to be eligible.
Rules and Logistics Q&A
- Grading stack reproducibility: FlopScope will receive further releases. If a change is meaningful enough to warrant re-evaluation, affected submissions will be re-evaluated. Final decisions will be based on the private evaluation round, run on a fixed version of FlopScope with full reproducibility details published.
- Non-determinism: Some non-determinism is attributed to “residual wall time,” but top submissions have optimized around it. The team will work to reduce this further before final prize decisions.
- Offline training: Training estimators offline on external Monte Carlo data (including publicly shared MLP moment datasets) is within the rules. There’s no way to track offline compute, so it doesn’t count toward anything computationally, but a more theoretical/mechanistic approach is seen as more likely to earn the algorithmic contribution prize than a purely learned/black-box estimator.
- Private rerun scoring: The designated final entry is re-evaluated on a fully fresh set of MLPs. No public-split score carries over. Exact evaluation details (e.g. number of MLPs) will be finalized closer to the time, after reviewing top submissions more closely.
- Why final-layer MSE only: Using only the final layer keeps the problem well-defined (estimating the output at one specific depth) rather than combining many separate problems (estimating every intermediate depth). Intermediate-layer data shown in the interface is for diagnostics only and isn’t part of scoring.
Technical Discussion
- Low-rank idea: As network depth increases, the effective rank of the activation covariance matrix decreases (eigenvalues become more skewed). ARC is internally exploring spending more compute on higher-variance directions rather than tracking every direction equally, as a way to make cumulant propagation scale better with depth.
- Cumulant propagation vs. quasi-Monte Carlo: The expectation is that cumulant propagation methods will probably beat quasi-Monte Carlo in the long run, with plans to build a strong quasi-Monte Carlo baseline to test whether a correctly implemented cumulant approach can outperform it. The low-rank refinement above is seen as the most promising direction, with less focus on exploring the space of Monte Carlo approaches since that’s considered less interesting from a research perspective.
- Higher-order Edgeworth series divergence: Pushing cumulant propagation to higher-order moments improves results initially but eventually diverges. ARC has explored Borel resummation as a possible fix but hasn’t fully solved it. They believe this issue matters more for very narrow networks (e.g. width 16) than for the current challenge parameters (width 256), where it shouldn’t be a blocker.
- Sparse matrix multiplication accounting: A forum question noted that flopscope doesn’t currently provided sparsity related cost savings, which are realistic on current hardware. (e.g. Nvidia GPU sparsity acceleration). ARC said this is worth considering for the cost model, without committing to a timeline.
Thank you again for your participation.
All the best,
Team AIcrowd