Status was changed after submission

Meta CRAG - MM Challenge 2025
#1

My submission of task 3 was successful before, but when I checked it today, it is “Failed”. Why is that?

################
Evaluation failed with exit code 1

[rank0]: Traceback (most recent call last):
[rank0]: File “<private_file>”, line 211, in
[rank0]: raise exc
[rank0]: File “<private_file>”, line 196, in
[rank0]: main()
[rank0]: File “<private_file>”, line 189, in main
[rank0]: serve()
[rank0]: File “<private_file>”, line 176, in serve
[rank0]: oracle_client.run_agent()
[rank0]: File “<private_file>”, line 269, in run_agent
[rank0]: self._agent_executor()
[rank0]: File “<private_file>”, line 323, in _agent_executor
[rank0]: result = self.execute(target_attribute, *args, **kwargs)
[rank0]: File “<private_file>”, line 173, in execute
[rank0]: return method(*args, **kwargs)
[rank0]: File “<private_file>”, line 123, in batch_generate_response
[rank0]: return run_with_timeout(
[rank0]: File “<private_file>”, line 159, in run_with_timeout
[rank0]: return fn(*args, **kwargs)
[rank0]: File “<private_file>”, line 327, in batch_generate_response
[rank0]: rag_inputs = self.prepare_rag_enhanced_inputs(
[rank0]: File “<private_file>”, line 227, in prepare_rag_enhanced_inputs
[rank0]: results = self.search_pipeline(<private_test_file>, k=<sensitive_data>)
[rank0]: File “/usr/local/lib/python3.10/site-packages/transformers/pipelines/base.py”, line 1431, in call
[rank0]: return self.run_single(inputs, preprocess_params, forward_params, postprocess_params)
[rank0]: File “/usr/local/lib/python3.10/site-packages/transformers/pipelines/base.py”, line 1439, in run_single
[rank0]: outputs = self.postprocess(model_outputs, **postprocess_params)
[rank0]: File “/usr/local/lib/python3.10/site-packages/cragmm_search/search.py”, line 186, in postprocess
[rank0]: return [
[rank0]: File “/usr/local/lib/python3.10/site-packages/cragmm_search/search.py”, line 187, in
[rank0]: {
[rank0]: TypeError: ‘<redacted_value>’ object is not a mapping
[rank0]:[W603 21:40:11.975361850 ProcessGroupNCCL.cpp:1476] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources.

1 Like
#2

This is because AIcrowd is re-running our submissions. Here is quote from @yilun_jin8

You don’t need to re-submit by yourself. We have queued all previous submissions for re-evaluation, and the re-evals will happen automatically.

The re-runs are failing because AIcrowd’s function search_pipeline() is returning None values now and this causes code to fail. (So instead of returning a python list with k items, it is just returning a single None value. But top k RAG should never return nothing).

TypeError: ‘NoneType’ object is not a mapping

2 Likes
#3

We believe that failures caused by changes in the API specification are outside our scope of responsibility. At the very least, the API interface should remain consistent before and after any updates.
Since we are participating in the competition while working full-time, it may be difficult for us to keep up with sudden API changes — which makes us a little sad.

We hope you will consider this to help make the competition even better.

#4

I have the same error. I think it’s a problem with search_pipeline(), not our own code

Traceback (most recent call last): File “<private_file>”, line 159, in run_with_timeout return fn(*args, **kwargs) File “<private_file>”, line 335, in batch_generate_response rag_inputs = self.prepare_rag_enhanced_inputs( File “<private_file>”, line 219, in prepare_rag_enhanced_inputs results = self.search_pipeline(<sensitive_data>, k=6) File “/usr/local/lib/python3.10/site-packages/transformers/pipelines/base.py”, line 1431, in call return self.run_single(inputs, preprocess_params, forward_params, postprocess_params) File “/usr/local/lib/python3.10/site-packages/transformers/pipelines/base.py”, line 1439, in run_single outputs = self.postprocess(model_outputs, **postprocess_params) File “/usr/local/lib/python3.10/site-packages/cragmm_search/search.py”, line 186, in postprocess return [ File “/usr/local/lib/python3.10/site-packages/cragmm_search/search.py”, line 187, in { TypeError: ‘NoneType’ object is not a mapping

#5

Meta released a fix for this issue and we are testing it out right now. We will soon re-evaluate the submissions with the fixed index.

1 Like