Hi Participants,
To provide more context to this important update:
The Missing Rate for a model refers to the rate at which the model refuses to answer a question by saying, “i dont know”. We have noticed that many of the current top solutions have a Missing Rate close to 90%, which is clearly unintended even if encouraged by the evaluation metric used for the current leaderboards. This is also very much against the spirit of the competition.
We would like to remind the participants that the final winners will not be determined by the Round 2 leaderboard scores, but by the result of the final Human Evaluation phase.
The Human Annotators understand the models they are evaluating are supposed to be meaningful QA systems, and we believe that brute optimization of the Missing Rate, will lead to lower scores in the final Human Evaluation phase.
Best,
Meta Organizers