Can we submit a stable solution to the competition?

Meta CRAG - MM Challenge 2025
#1

@yilun_jin8 @jyotish
Hello admin.
My understanding is that we can’t submit right now. is that correct?

  1. v0.6 Search Pipeline Problem
    Why Does Private Web Search v0.6 search_pipeline() return None?
    If we submit stability, competition host will fix this search api.

  2. Storage Problem
    host is expanding storage, I didn’t know instance storage already fixed or not.
    Why Submission #286586 failed?

Anyway, Since the competition period has been shortened, please consider compensatory measures such as allowing more submissions.

Thanks.

1 Like
#2

Hi @tereka,

The issue that cragmm_search index v0.6 is returning None values has been directed to Meta. We will work with them to ensure that it is properly solved. The API and indices are developed by Meta, and I am afraid that we cannot solve it from the side of aicrowd.

The issue of host storage has been addressed. We have expanded the available storage to ensure that participants can use ~250 GB for their submission files.

We will also discuss possible compentations with the organizers from Meta such as extensions and increasing submission quota.

Sorry for all the confusion caused. .

#3

Hi @yilun_jin8
Thank you for reply.
I understand situation at now.

For your information, we can submit Single-Source Augmentation,
but we cannot submit others because of cragmm_search index v0.6 None Exception. Is that correct?

#4

I think yes, because only the web search index is updated to v0.6, but the image search index is still 0.5.

#5

Thanks, I completely understand.
I’m waiting for v0.6 fixed version about Multi Turn/Multi Source.

#6

@yilun_jin8
Sorry to jump in, but if possible, could the issue be addressed by increasing the number of allowed submissions instead of extending the deadline? Alternatively, could failed submissions be excluded from the submission count?

We have planned our schedule based on the originally set deadline, and repeated extensions may disadvantage at least some participants.

#7

Is it possible to extend deadline?

#8

@yilun_jin8
Hello.I would like to confirm about these problems.

  1. When will the API be fixed?, I will submit my solution in Phase2/3
  2. When will host announce about possible compensation?
#9

@tereka

The API issues are already fixed and we are re-evaluating all the submissions. We will post an update once the re-evaluations are completed. You can submit your solutions to all the tasks.

#10

@jyotish
Thank you for reply.
I’m waiting for host announcement about re-evaluation process.

how about this?

When will host announce about possible compensation?

#11

The organizers have agreed to increase the quota of submission. However, the timeline won’t be extended. We will announce it pretty soon.

1 Like
#12

We have added an additional 10 failed submissions per week as a compensation.