Potential bug with the evaluation metrics for Track 1

wangzhiyu918 · March 16, 2025, 2:38pm

Hi,

We have identified a potential issue with the evaluation metrics for Track 1.

According to the official rules, the primary evaluation metric for Track 1 is AV_ALIGN. However, the AV_ALIGN score displayed on the leaderboard appears to be inconsistent with the score shown on the submission page. For instance:

The current top-ranked AV_ALIGN score on the leaderboard is 0.418.

However, on the submission page here, the AV_ALIGN score is 0.357.

We suspect that the scores for AV_ALIGN and TA may have been inadvertently swapped, which could lead to confusion among participants.

Could you please look into this issue?

Thank you for your time and assistance.

Best regards

snehananavati · March 17, 2025, 5:50am

We are reviewing it and will get back to you.

wangzhiyu918 · March 17, 2025, 6:12am

Thanks for your attention.

wangzhiyu918 · March 18, 2025, 5:02pm

@snehananavati Anything new? The deadline is coming

aicrowd_team · March 18, 2025, 8:21pm

@wangzhiyu918 : This is resolved now.

kartana · March 19, 2025, 2:26am

It’s quite funny that the deadline is only 6 days away, and I spent 2 weeks just to improve an arbitrary score =)) I don’t even know what the previous score was.
You should give me an explanation, right? @snehananavati @aicrowd_team

wangzhiyu918 · March 19, 2025, 3:44am

@snehananavati @aicrowd_team Any plans to extend the deadline? We request for more time to enjoy this challenge.

OwO · March 19, 2025, 8:59am

@aicrowd_team Not work for new submission.
This ranking is problematic. Even the evaluation method is seriously problematic. Deadline is approaching but it is causing a lot of confusion for competitors.

OwO · March 19, 2025, 9:05am

@wangzhiyu918 I see your new submission same the baseline. And do you see the leaderboard still show the old AV-Align? Could you confirm it?

wangzhiyu918 · March 19, 2025, 9:15am

@OwO Yes, this problem still exists.

@aicrowd_team Please check this again

wangzhiyu918 · March 19, 2025, 11:00am

@snehananavati We sincerely request additional time to further improve our solutions.

aicrowd_team · March 19, 2025, 1:34pm

Dear Participants,

We wanted to inform you that the issue regarding the scoring system has now been resolved. The previous fix did not fully propagate due to a caching issue, which has been successfully addressed.

Regarding the challenge deadline, we had thorough discussions with the Sony team. While we would have greatly preferred to offer an extension, external factors beyond our control prevent us from doing so. Therefore, the previously announced deadline of March 25th remains unchanged.

Thank you for your understanding and continued participation.

Best regards,
AIcrowd Team