Key Updates: Local End-to-End Evaluation Script (Docker), Temporal Track Dependency Fix, and Dockerfile Optimizations

Sounding Video Generation (SVG) Challenge 2024
#1

:arrows_counterclockwise: SVG Challenge 2024 Starter Kit Updates

Hey everyone! We’ve made some important updates to the starter kits to improve your development experience.

:warning: Important: Update Your Fork

Please pull the latest changes into your fork to get these improvements:


# Add the upstream remote if you haven't already

git remote add upstream git@gitlab.aicrowd.com:aicrowd/challenges/sony-sounding-video-generation-challenge-2024/svg-2024-spatial-track-starter-kit.git

# Fetch the latest changes

git fetch upstream

# Merge them into your main branch

git merge upstream/main

For Temporal track participants, use git@gitlab.aicrowd.com:aicrowd/challenges/sony-sounding-video-generation-challenge-2024/svg-2024-temporal-track-starter-kit.git as the upstream.

:whale: New Docker Testing Script

We’ve added a docker_build_run.sh script has been added to both the starter kits that lets you test your submission exactly as it would run on our evaluation servers. This script:

  • Builds the docker image locally

  • Runs the full evaluation pipeline on the sample dataset

  • Helps catch any runtime issues before you submit

:zap: Docker Build Optimization

The Dockerfile has been optimized to be more efficient:

  • No longer copies the entire repository into the image

  • Significantly reduces build time and image size

  • Makes iterating on your submissions faster

:wrench: Temporal Track Fix

For participants in the Temporal track:

How to Use the New Docker Script

Simply run:


./docker_build_run.sh

This will give you the most accurate preview of how your submission will perform on our evaluation servers. We urge you to ensure that this runs successfully before you make a submission.

Best,
AIcrowd Team

1 Like
💬 Feedback & Suggestions
#2

@aicrowd_team
Dear, aicrowd_team.
I think there is a bug in an evaluation score after this update.
Before the update, the scores of spatial_svgc_err and spatial_av were not the same.
However, after it, the scores are getting the same.
Could you check this bug, please?
Thank you very much.

Bug picture

스크린샷 2024-12-28 오전 8.28.34
스크린샷 2024-12-28 오전 8.28.34448×936 20.6 KB

#3

@kcy4 : This has been resolved now. Thanks for the note.

1 Like
#4

@aicrowd_team : Thank you for the speed fix.

#5

I’m unable to clone the starter kit via HTTPS for submission, as it’s blocked. Here’s the command I’m using:

!git clone https://{username}:{personal_token}@gitlab.aicrowd.com/samtech/svg-2024-temporal-track-starter-kit.git

But I get this error

Cloning into 'svg-2024-temporal-track-starter-kit'...
remote: Git access over HTTP is not allowed
fatal: unable to access 'https://gitlab.aicrowd.com/samtech/svg-2024-temporal-track-starter-kit.git/': The requested URL returned error: 403

Can HTTP(S) cloning be enabled so we can run submissions in online notebooks without downloading model weights locally?

Thanks!

#6

Dear, @aicrowd_team.
I have a question.
Is it okay to ask you to review the competition score(or metric) for the ‘Temporal Alignment Track’.

P.S. Thank you for the great competition. This competition is really interesting in terms of its subject and difficulty for me.

#7

@samtech: How about using ssh?

#8

@samtech : Please use git clone over ssh. git over http is disabled on gitlab.aicrowd.com as clones over HTTP for large models are extremely inefficient with git-lfs, and we do not plan to enable them anytime soon.

@kcy4 : Can you please check now ?

#9

@aicrowd_team : I checked the fix. Thank you.