💬 Feedback & Suggestions

Sounding Video Generation (SVG) Challenge 2024
#1

We are constantly trying to improve this challenge for you and would appreciate any feedback you might have! :raised_hands:

Please reply to this thread with your suggestions and feedback on making the challenge better for you!

  • What have been your major pain points so far?
  • What would you like to see improved?

All The Best!

#2
#3
#4

I submitted my baseline but the progress is stuck in the validation phase. Can you help me check this?

#5

@wangzhiyu918 : This should be resolved with th instructions in this post. Best of luck.

#6

@aicrowd_team Hello, I have submitted in phase 2, but the leaderboard hasn’t updated yet. Could you please to check this?

#7

@wangzhiyu918 : we see your submission has already been evaluated now. Best of luck.

#8

@aicrowd_team Yes, however, it has not shown in the leaderload of phrase2

#9

@aicrowd_team My submission has been evaluated sucessfully, however, it has not shown in the leaderboard of phrase2.

#10

Hi, The issue is now fixed. You should be able to see your submission on the leaderboard now.

#11

@snehananavati Could you check for me in task temporal-alignment-track. I can’t submit my code to gitlab because I got this error (I clicked and accepted all the rules)

sub hash: 6ded30585b5f2672b9c3c07888d7dce4a25d9dc2

#12

@snehananavati Could you check for me in task temporal-alignment-track. I can’t submit my code to gitlab because I got this error (I clicked and accepted all the rules)

Submission failed : You have not qualified for this round. Please review the challenge rules at www.aicrowd.com

#13

Dear, @aicrowd_team.
Thank you for this great competition.
I have two questions.

  1. What is the exact eval score?
  • According to the leaderboard, the eval metric seems like it is av_aglign, which is the same as TA for #280900, but is the same as AV_ALIGN for #280380 for me. Which one is the correct one?
  1. Can we choose two submissions at the end of the competition?

Best regards

#14

Hello, Can you please specify the track for your first query?
Edit: As for your second query, at the end of the competition (subjective evaluation), the best system for each participant is used, even if three systems from the same participant are ranked 1st-3rd.

#15
  1. It’s for [Temporal Alignment Track]
  2. I didn’t understand the meaning of “even if three systems from the same participant”. The competition is in phase 2(stage) now. Do you mean that the best system(submission) for both phase 1 and phase 2 will be used for the final? There was warm up stage, but we can’t see the stage(warm up) result now.
#16

Here’s a response from the organiser:

  1. The answer is av_align. We’d like to refer you to the following challenge rules for details.
    AIcrowd | Sounding Video Generation (SVG) Challenge 2024 | Challenge_rulesThe following explanation is an excerpt from the challenge rule page.

We use the AV-Align as the main metric for ranking and the CAVP score as the secondary metric to break ties. The other four metrics are used to exclude entries that provide low-quality data from the ranking. Specifically, if the score of the submitted model does not exceed the threshold value in any one of these four metrics, the model is excluded from the ranking. The threshold is set as follows: 2.0 for FAD, 900 for FVD, 0.25 for LanguageBind text-audio score, and 0.12 for LanguageBind text-video score.

  1. We might have misunderstood your question, but let us clarify what we meant. As explained in the challenge rules,

The top entries in the final leaderboard will be assessed by human evaluation, and the award winning teams will be selected based only on the results of this subjective evaluation.

We plan to assess top 10 models chosen according to av_align. In the 2nd phase, you can submit multiple systems, However, even if you occupy the leaderboard from the 1st place to 10th in the end, we will assess only the top-1 model of yours and pick out 9 models from other participants so that we can reward as many participants as possible.

Hope this answers your questions.

1 Like
#17

@snehananavati @aicrowd_team Hello, for Track 2, would it be possible to first generate a video using an unconditional video generation model and then synthesize the corresponding audio using a video-to-audio model?

#18

@snehananavati @aicrowd_team
Hi，I hope this message finds you well.

I am writing to kindly inquire about the status of the results for the [Sounding Video Generation (SVG) Challenge 2024], which concluded over a month ago. I understand that preparing and reviewing submissions can be a time-consuming process, and I truly appreciate all the effort that has gone into organizing the event.

Could you please let me know when we can expect the official announcement of the results?

Best regards.

1 Like
#19

Hello,

The organisers at Sony are currently conducting the final round of human evaluation. As per the challenge rules, the top entries on the final leaderboard will be assessed through human evaluation, and the winning teams will be selected based on the results of this subjective assessment.

We are awaiting the outcome from the organisers and will share an update as soon as we receive it.

Thank you for your patience.

2 Likes