Vulnerability

Music Demixing Challenge ISMIR 2021
#1

What happens if a participant does something like this in the submission code?

import os
os.environ['INFERENCE_PER_MUSIC_TIMEOUT_SECONDS'] = '250' #240
3 Likes
#2

Hi @agent,

We take care of approaches like the one you mentioned i.e.

os.environ['INFERENCE_PER_MUSIC_TIMEOUT_SECONDS'] = '250'

and so on by using a different evaluation class & env variables instead of the one provided in the starter kit.

But at the same time,

I agree with enough motive and context someone can fool the system by:

  • borrowing more time to a longer song from smaller songs’ timeout.
  • monkey patching the evaluation phase with their custom commands, etc to do anything in global timeout time.

NOTE: We still have global timeout i.e. initial_setup + (per_song * total_songs) + small_buffer which isn’t accessible by user code.

We are continuously improving and would be switching to a different model, in which even borrowing of the time isn’t possible (client-server model and no access to future songs :wink:) hopefully by Round 2.

In case you wish to share any ideas, you are welcome to share them with us here on Discourse or privately on devops@aicrowd.com.

4 Likes