🚨 IMP: Phase 2 Announcement

Meta Comprehensive RAG Benchmark: KDD Cup 2
#1

Phase 2 Laucnh Updates: Comprehensive RAG Benchmark (CRAG) 2024

Hello everyone,

Welcome to Phase 2 of the CRAG: Comprehensive RAG Benchmark. Please carefully read the updates below.

Eligibility

  • Teams: All teams with at least one successful submission in Phase 1 can enter Phase 2.

Key Updates for Phase 2

  1. Time Limit: The time limit per example is 30 seconds.
  2. Submission Limit: The submission limit for Phase 2 is 10 per week.
  3. MockAPI Update: The MockAPI has been updated to include a private Knowledge Graph.
  4. Leaderboard: 20% of the private test set will be used to run the leaderboard (same as Phase 1b).
  5. Final Evaluation:
    • As soon as this phase ends, a final exhaustive evaluation will be conducted on the full Phase 2 test set on a submission of your choice (defaulting to your best submission on the leaderboard).
    • The top submission chosen by the team will be used for the final evaluation.
  6. Submission Deadline: The last date to make a submission is June 20, 2024, at 23:55 UTC.

All the best,

Team CRAG 2024

1 Like
#2
#3

Is the leaderboard set incorrectly? The order is reversed.

image
image2112×752 104 KB

#4

Did anyone encounter this problem in phase 2 when submitting? I didn’t do any parallel submissions. Who should I contact to resolve this issue?

Submission failed : You have exceeded the allowed number of parallel submissions. Please wait until your other submission(s) are graded.
submission_hash : b5f132be7805964aa45a45bdbdfb8ed26cd8bea3

#5

I’ve encountered this. I had one submission which is completed but not correctly tagged, which seems the resources are not released correctly. So that submission may be seen as a parallel submission. I don’t know how to deal with it. I’ve sent an email to help@aicrowd.com but got no responses. Do you have similar (wrongly tagged) uncompleted submissions? There are some other discussions about this: Parallel Submission Limit Reached. But no official responses until now. @aicrowd_team Could anyone help with this?

#6

Yes, I have a similar situation. I tried to manually delete the incorrectly tagged issue (submission) in gitlab, but it didn’t work. The incorrectly tagged submission happened 2 weeks ago and the first time I encountered this “parallel submission” problem was on June 4th. I couldn’t make any successful submission from that day.

#7

same problem, I also email the help@aicrowd.com but no responses

#8

Seems no one in Meta cares about this competition, lol. There is rarely any official staff in the discussion forum to solve problems.

#9

We are confirming that the parallel submission limit has now been removed.
The sorting of the leaderboard is now marked correctly to Descending.

#10

Thanks for your working. The LB for task-2 is still incorrect. AIcrowd | Meta KDD Cup 24 - CRAG - Knowledge Graph and Web Retrieval | Leaderboards.

图片
图片2652×1066 204 KB

#11

what does api update mean, is the test environment updated to the new api?

#12

Didn’t say there were 10 submissions a week in phase two? Why have I only submitted four times and now there is no opportunity to submit?