🚨 [Phase 2 Now Open] Important Updates and Announcements

Meta CRAG - MM Challenge 2025
#1

Dear participants,

Phase 2 of the Meta CRAG-MM Challenge 2025 is now live. Please find below the final set of updates based on the v0.1.2 release of the dataset and evaluation.

:arrows_counterclockwise: Key Updates for Phase 2

  • v0.1.2 Released: Includes updated datasets, mock APIs, and schema changes.
  • Submissions now open, based on v0.1.2.
  • Validation and public test sets remain accessible after the challenge.
  • Leaderboard now reflects performance on the private test set.
  • Submission cap: Up to 10 submissions per task per week.
  • Egocentric images are downsampled to 960×1280 for evaluation to reflect real-world constraints. Standard image resolution remains unchanged.
  • Submission deadline extended to 14 June 2025; team freeze deadline extended to 1 June 2025.

:warning: Please note: The rules have been updated to reflect these changes. To continue submitting in this round, you must visit the challenge page, click “Participate”, and accept the updated rules.

:package: What’s New in v0.1.2

  • Cleaner data: Removed low-quality or ambiguous image–QA pairs.
  • Expanded volume: More samples added across all three tasks.
  • Bug fixes: Minor corrections (e.g. consistent domain values).
  • Improved web search: Upgraded to BGE text encoder.

:hammer_and_wrench: Schema Update & Loader Compatibility

  • Schema upgraded: Single-turn and multi-turn datasets now follow a dict-of-columns format. No fields have been removed or renamed.
  • If you’re using crag_batch_iterator.py, no changes needed — it auto-detects schema layout.
  • If you stream datasets directly from Hugging Face, minor updates are required (usually 2–3 lines). See the [schema update post] for examples and a minimal helper function.
  • The iterator has been patched to:
    • Handle both list and dict layouts
    • Auto-download missing blobs and resize to 960×1280
    • Provide clearer error messages for missing fields
  • Updated dependencies required: Ensure your environment uses:
    • torch > 2.6
    • vllm < 0.8
    • cragmm-search-pipeline >= 0.5.0

Click here for more details

Please pull the latest changes, update your environment, and continue training. If you encounter any issues, run the helper function from the post and share the output — we’re here to help.

We look forward to your submissions in this new round.

#2
#3

Hi,

I tried to install using the requirements given from the gitlab files but got this error. Also it would be helpful to update local evaluation.py to use both validation and public_test splits. Thank you.

image
image2727×1061 214 KB

Summary

This text will be hidden

1 Like
#4

same error here. Please update this.

#5

Hello @harshavardhan3 @fersebasIn

Recent releases of vllm are currently affected by an undefined-symbol error (see the discussion in this issue). In our internal testing, the vllm 0.7.x series functions correctly.

That said, vllm 0.7.x requires torch 2.5.1, and this particular Torch build cannot load CLIP model weights due to the security advisory CVE-2025-32434. We are working on a more robust solution, but the following interim procedure will allow you to run local tests—including both the validation and public_test splits—without encountering these issues:

  1. In requirements.txt, set

    vllm==0.7.3

  2. Remove any explicit version constraint for Torch in the same file.

  3. Install dependencies:

    pip install -r requirements.txt

  4. Reinstall Torch with compatible versions:

    pip install torch==2.6 torchvision==0.21.0

After completing these steps, local_evaluation.py should execute successfully on both data splits.

Please let us know if you encounter any further difficulties.

#6

Thanks, we will try this solution.