🚨 IMP: Phase 2 Announcement

Phase 2 Laucnh Updates: Comprehensive RAG Benchmark (CRAG) 2024

Hello everyone,

Welcome to Phase 2 of the CRAG: Comprehensive RAG Benchmark. Please carefully read the updates below.


  • Teams: All teams with at least one successful submission in Phase 1 can enter Phase 2.

Key Updates for Phase 2

  1. Time Limit: The time limit per example is 30 seconds.
  2. Submission Limit: The submission limit for Phase 2 is 10 per week.
  3. MockAPI Update: The MockAPI has been updated to include a private Knowledge Graph.
  4. Leaderboard: 20% of the private test set will be used to run the leaderboard (same as Phase 1b).
  5. Final Evaluation:
    • As soon as this phase ends, a final exhaustive evaluation will be conducted on the full Phase 2 test set on a submission of your choice (defaulting to your best submission on the leaderboard).
    • The top submission chosen by the team will be used for the final evaluation.
  6. Submission Deadline: The last date to make a submission is June 20, 2024, at 23:55 UTC.

All the best,

Team CRAG 2024

1 Like

Is the leaderboard set incorrectly? The order is reversed.

Did anyone encounter this problem in phase 2 when submitting? I didn’t do any parallel submissions. Who should I contact to resolve this issue?

Submission failed : You have exceeded the allowed number of parallel submissions. Please wait until your other submission(s) are graded.
submission_hash : b5f132be7805964aa45a45bdbdfb8ed26cd8bea3

I’ve encountered this. I had one submission which is completed but not correctly tagged, which seems the resources are not released correctly. So that submission may be seen as a parallel submission. I don’t know how to deal with it. I’ve sent an email to help@aicrowd.com but got no responses. Do you have similar (wrongly tagged) uncompleted submissions? There are some other discussions about this: Parallel Submission Limit Reached. But no official responses until now. @aicrowd_team Could anyone help with this?

Yes, I have a similar situation. I tried to manually delete the incorrectly tagged issue (submission) in gitlab, but it didn’t work. The incorrectly tagged submission happened 2 weeks ago and the first time I encountered this “parallel submission” problem was on June 4th. I couldn’t make any successful submission from that day.

same problem, I also email the help@aicrowd.com but no responses

Seems no one in Meta cares about this competition, lol. There is rarely any official staff in the discussion forum to solve problems.

We are confirming that the parallel submission limit has now been removed.
The sorting of the leaderboard is now marked correctly to Descending.

Thanks for your working. The LB for task-2 is still incorrect. AIcrowd | Meta KDD Cup 24 - CRAG - Knowledge Graph and Web Retrieval | Leaderboards.

what does api update mean, is the test environment updated to the new api?

Didn’t say there were 10 submissions a week in phase two? Why have I only submitted four times and now there is no opportunity to submit?