Looking at the revised code, it appears that the non-ego images are being resized. Shouldn’t the condition be the other way around?
1 Like
You are correct. The code is backwards. When image_url is None we have egocentric image and it should be rescaled. When image_url is not None then we have non-egocentric image and should not be rescaled.
If we correct the code in crag_batch_iterator.py in our GitLab, will submission inference use our crag_batch_iterator.py or does it use another one?
And if submission uses our crag_batch_iterator.py, what prevents us from changing the ground truth to “black” for every question and then predicting “black” for every question to achieve perfect LB score?
Good catch on the resize logic — the condition was indeed inverted. The starter kit has been patched once more to reflect the correct behaviour.
To clarify the doubts around evaluation pipeline:
-
Local vs. leaderboard: The evaluation server does not run the
crag_batch_iterator.pyshipped in the starter kit. It loads only your agent class and the modules that your agent explicitly imports. The rest of the starter code is purely for local convenience and mirrors our internal setup except for items such as the LLM-judge prompt and progress logging. -
Data integrity: Ground-truth labels reside exclusively on the evaluation server and are never exposed to user code. Consequently, altering any local file—iterator or otherwise—cannot modify the answers used for scoring. Submissions that alter the ground truth and predict “black” for every example will be graded against the genuine ground truth on the server and will score accordingly.
@jyotish @Chris_Deotte
Thank you very much for your prompt response and for updating the code.
I also understand now that the local and leaderboard evaluation scripts operate independently.
Thanks!
I was reviewing the code and just wanted to double-check something.
Would the intended behavior perhaps be this?
if image_url is None:
image = image.resize((960, 1280))
Instead of:
if image_url:
image = image.resize((960, 1280))
Please let me know if I’m misunderstanding anything!
1 Like
You are right! 
1 Like