Thanks a lot for sharing this!
If it is ok, I would like to share a few thoughts:
-
for write-ups, if I was an organizer I would have imposed a pages number cap. Ideally, max 10 pages (with standard characters). This would incentivize people into writing more of their own, and would make easier to review write ups. I myself had difficulty in reading reports of 20-50 pages (though there is a selection bias given these are the public ones, and I can not generalize on the rest).
- popular LLMs output natural language text, except Gemini I suppose, is currently quite hard to read. I am guilty myself of using them and now that I am more relaxed I can see they are not great in exposing ideas. Hence giving a cap in pages might incentivize some people in putting more effort into the content and making it more personal.
-
for competition, I am not sure the current competition metric is helping solve the challenge itself. I am sure you and the other organizers thought deeply about it. But I think a less difficult setting, and cutting maximum flops might help steering people into tackling the core of the problem (though perhaps someone might try to go hard on training some prediction system). Or even making sampling based techniques too expensive. Phase 1 surfaced incentives that might have made competitors shy into pursuing more mechanistic approaches.
- This is obviously too late to state given Phase 2 is starting soon.
Last but not the least: it would be really, really appreciated if we had less LLM content on the forum itself!