🚨 Submission Selection Deadline: 23rd June 2025, 12:00 UTC (noon)

Meta CRAG - MM Challenge 2025
#1

Hello everyone, and thanks again for participating in the Meta CRAG-MM Challenge 2025.

We’re now collecting final submission selections for each track. Please fill out the form here:
:point_right: https://forms.gle/51qr1ngiqagKaMQ4A

Please note:

  • You may select up to two submissions per team per track
  • Submissions must be eligible for the respective track (i.e. already on the leaderboard for that track)
  • If more than two submissions are listed, we’ll use the last two
  • If no form is submitted, we’ll default to your top two leaderboard submissions

:bust_in_silhouette: Individuals: Please fill out the form only once. If submitted multiple times, we’ll use the latest entry
:busts_in_silhouette: Teams: If multiple team members submit, the Team Organizer’s latest entry will be used

— Meta CRAG-MM Challenge Team

#2

How to solve the situation where the Submission #289819 score has been given, but the status is still in submitted? And the given score is not update on the leaderboard… AIcrowd | Multi-turn QA | Submissions #289819

#3

@aicrowd_team @yilun_jin8 @Jiaqi Can you see this post for me? I think these two submissions have some system problems and need to be reruned.

#4

Hi. When we select two, what do you do with the two? Will you human evaluate both of the two and pick the one that has the best score? (i.e. of the two selected per task, which one of the two will eventually be used as our team’s final submission per task?)

2 Likes
#5

I think it now shows ‘graded’?

#6

Hello,

I am about to submit my entries, but I noticed a possible mistake in the submission form.

Currently, the form displays:

  • Task 2: Multi-turn QA
  • Task 3: Multi-source Augmentation

However, I believe the correct task names should be:

  • Task 2: Multi-source Augmentation
  • Task 3: Multi-turn QA

Could you please confirm which is correct and update the form if necessary?
Thank you very much!

#7

Hello, thanks for flagging this. You are right, it should be Task 2: Multi-source Augmentation and Task 3: Multi-turn QA. It is fixed now.

1 Like
#8

@snehananavati @yilun_jin8 @jyotish Before we can submit the form, we need to know why we are choosing two. Could you please explain how the two will be used? For example

  • If we submit two, you will use the one with less missing responses.
  • If we submit two, you will choose the code that you like best
  • If we submit two, you will human evaluate both and choose the one with best score
  • If we submit two, you will choose the one which runs the fastest
  • etc, etc, etc

Please respond quickly so we have time to pick our two final submissions before selection deadline. Thank you!

3 Likes
#9

@Chris_Deotte

Only one submission per team per track will be considered in human grading. We will choose the one with higher auto-eval score while satisfying the missing rate constraint.

We’d recommend teams to submit another solution in case their best solution (by truthfulness score) has high missing rate (e.g. >80% for task 2 and 3).

1 Like