Request for investigation into the cause of the error

Meta CRAG - MM Challenge 2025
#1

hi @aicrowd_team
I would like you to investigate the cause of the failure of the following submit:

Task2

Task3

※ I’m running into a strange issue with timeouts and I’m not sure if it’s my code or the system. I’m getting inconsistent results: sometimes my submission passes, but other times it times out, even with almost no changes to the code.

Could you please take a look and help me figure out what’s going on?

1 Like
#2

Hello @NineGates

We are re-evaluating 289818 (seems like some network error caused it to fail with the error “failed to load artifacts”) and 289826 (evaluation didn’t start).

We can’t re-evaluate the rest as they failed due to timeout errors. Please note that it’s not the average prediction time we look at it, there is hard constraint on per step timeout. One step that violates this can fail the submission.

  • 289762 violated the overall timeout.
  • 289784 failed due to a per step timeout

2025-06-18 08:18:43.290 INFO main:run_with_timeout:154 - Running batch_generate_response with timeout 80
2025-06-18 08:20:28.554 INFO main:run_with_timeout:164 - Executed batch_generate_response in 105.26403450965881 seconds

  • 289820 failed due to a per step timeout

2025-06-18 07:06:50.363 INFO main:run_with_timeout:154 - Running batch_generate_response with timeout 80
2025-06-18 07:08:14.969 INFO main:run_with_timeout:164 - Executed batch_generate_response in 84.60583829879761 seconds

1 Like
#3

@jyotish @yilun_jin8 @aicrowd_team
Thank you for your support throughout the competition period.