Hi,I want to know what the following error message means

Meta Comprehensive RAG Benchmark: KDD Cup 2-9d1937
#1

rank0: Traceback (most recent call last):

rank0: File “<private_file>”, line 211, in

rank0: raise exc

rank0: File “<private_file>”, line 196, in

rank0: File “<private_file>”, line 189, in main

rank0: File “<private_file>”, line 176, in serve

rank0: File “<private_file>”, line 195, in run_agent

rank0: raw_response, status, message = self.process_request(

rank0: File “<private_file>”, line 101, in process_request

rank0: “data”: self.route_agent_request(

rank0: File “<private_file>”, line 131, in route_agent_request

rank0: return self.execute(target_attribute, *args, **kwargs)

rank0: File “<private_file>”, line 144, in execute

rank0: return method(*args, **kwargs)

rank0: File “<private_file>”, line 124, in batch_generate_response

rank0: return run_with_timeout(

rank0: File “<private_file>”, line 159, in run_with_timeout

rank0: return fn(*args, **kwargs)

rank0: File “<private_file>”, line 329, in batch_generate_response

rank0: rag_inputs = self.prepare_rag_enhanced_inputs(

rank0: File “<private_file>”, line 219, in prepare_rag_enhanced_inputs

rank0: results = self.search_pipeline(<sensitive_data>, k=<sensitive_data>)

rank0: File “<private_file>”, line 1379, in call

rank0: return self.run_single(inputs, preprocess_params, forward_params, postprocess_params)

rank0: File “<private_file>”, line 1386, in run_single

rank0: model_outputs = self.forward(model_inputs, **forward_params)

rank0: File “<private_file>”, line 1286, in forward

rank0: model_outputs = self._forward(model_inputs, **forward_params)

rank0: File “<private_file>”, line 143, in _forward

rank0: raise ValueError(

rank0: ValueError: <redacted_value>

#2

Could you please give me your submission ID?

#3

Also, it seems that you are posting questions in the Meta CRAG KDD Cup 2024 challenge. Please post to the Meta CRAG-MM 2025 challenge (AIcrowd | Meta CRAG - MM Challenge 2025 | Challenges) instead next time.

#4

My submission ID is # 284443

#5

{
“challenge_id”: “single-source-augmentation”,
“gpu”: true,
“hf_models”: [
{
“repo_id”: “meta-llama/Llama-3.2-11B-Vision-Instruct”,
“revision”: “main”
},
{
“repo_id”: “your-org/your-model”,
“revision”: “your-custom-revision”,
“ignore_patterns”: “*.md”
}
]
}
Do I need to include the dataset in aicrowd.json when submitting?
For example,
web_hf_dataset_id=“crag-mm-2025/web-search-index-validation”,
image_hf_dataset_id=“crag-mm-2025/image-search-index-validation”,

#6

The error is caused by the fact that you are trying to use web search in task 1.

In task 1 ‘single-source augmentation’, we only enable image search, so this error is expected.

#7

You don’t need to include the indices. They are instantiated by the evaluator.

#8

but for this submission the “challenge_id” is “multi-source-augmentation” not task 1

#9

No, I can clearly see that this submission goes to task 1.

image
image2718×200 44.1 KB

This is also reflected in your aicrowd.json.
image
image2340×1442 273 KB

#10

thanks! I will check this

#11

Hello, please ask the following questions:
Q1: According to the Truthfulness Score scoring rules in the single-turn QA in the figure, shouldn’t the higher the score, the better the performance? Why is the current ranking higher for lower scores? How to understand this score?

1747223929971
17472239299711276×354 47.5 KB

Q2: What conditions must the Potential winners mentioned in the figure meet? Is it the top 10 for each task?

1747223990507
1747223990507851×202 23 KB

Q3: About # initiate both image and web search API
search_pipeline = UnifiedSearchPipeline(
image_model_name=“openai/clip-vit-large-patch14-336”,
image_hf_dataset_id=“crag-mm-2025/image-search-index-validation”,
text_model_name=“sentence-transformers/all-MiniLM-L6-v2”,
web_hf_dataset_id=“crag-mm-2025/web-search-index-validation”,
)
Are there any restrictions on the use of models in this part, or can only the official models be used?

Thank you for your time！

#12

First of all, you are raising the question in the Meta CRAG 2024 Challenge. Please raise the questions in the Meta CRAG-MM 2025 Challenge (AIcrowd | Meta CRAG - MM Challenge 2025 | Challenges) if you have further questions.

A1: If you are looking at the leaderboard of Task 1, you will see that the ranking is correct — the higher the scores, the higher the rankings.

image
image2826×1204 249 KB

A2: I actually do not have a concrete idea about this, but I suppose it will be stricter than top 10 for each task.

A3: The search pipeline will be provided to you during the initialization of your agent (for example, here). You won’t need to initialize it yourself. Moreover, even if you can, we recommend against it, because the embeddings in the vector databases are generated with these models. If you use other models, the embeddings may not be in the same feature space, thus compromising retrieval quality.

#13

thanks for your time！

#14

Hello, for the Q3, I would like to ask for more information. What I want to know is whether we can rewrite the UnifiedSearchPipeline class, which means that the embedding model remains unchanged and we rewrite the RAG strategy ourselves. If it can be rewritten, will the downloaded cragmm_search package still be the previous one when uploading and submitting the evaluation, or will it run my local code? If the methods in the UnifiedSearchPipeline class cannot be rewritten, then as a Multi-source Augmentation track, the contestants seem to be unable to make changes in the RAG part, which also means that we cannot make too many changes to this track, so what can we do in this track?

1747725725585
17477257255851153×946 48.5 KB

#15

Hi,

During actual evaluation, we will nonetheless provide the default version of UnifiedSearchPipeline initialized by the evaluator. Local codes won’t be effective.

You can try to improve the generation model (e.g. make sure it says “I don’t know” upon uncertainty), the pre-processing of RAG, etc. You can check out the following posts to see how last year’s top teams achieve good scores.

#16

Thanks! Is fine-tuning the VLM model( e.g. Llama 3.2 11B) allowed?

#17

Yes. That is allowed.