Build image failed with "pod deleted during operation"

Music Demixing Challenge ISMIR 2021
,
#1

Hi, my submission today was failed at the build image stage. The logs didn’t show any error but this line of message pod deleted during operation. Is this a server side problem?

My submission:
http://gitlab.aicrowd.com/yoyololicon/music-demixing-challenge-starter-kit/-/issues/16

1 Like
#2

I submitted again and meet the same problem.
https://gitlab.aicrowd.com/yoyololicon/music-demixing-challenge-starter-kit/-/issues/17
Seems like the runner was killed with no reason.

#3

Hi @yoyololicon,

The issue with your submissions has been identified and fixed now.
It was related to a large repository size causing RAM issues during docker build.

In case you face docker related issues in the future, please let us know. :smiley:

2 Likes
#4

I made 3 submissions at once. All of them failed with message:

pod deleted during operation

Then I submitted one of them and it was evaluated successfully.

@shivam, should participants refrain from parallel submissions?

#5

Hi @agent,

No, it looks like an infrastructure issue, let me check and get back on it.

You should be able to submit parallelly ideally once this is fixed.

#6

Hello @shivam

I made 4 submission at once. Results:

#149836 ok
#149838 pod deleted during operation
#149839 ok
#149840 Evaluation timed out

It would be great if the issue was fixed before the end of the challenge.

My previous submissions is probably not available to you because I recreated my repository.

#8

Hi @shivam,

My recent submissions meet this “pod deleted during operation” problem too. The image-building process was killed out of nowhere. Could you help me fix this issue? Or can the submission with this kind of error exclude from the counting of total submission time? Because this error is not caused by us and the submission time for failed submission is also limited per day (10 perhaps).Thanks a lot!

AIcrowd Submission Received #151635 - v5.6 (#3) · Issues · HaoheLiu / music-demixing-challenge-starter-kit · GitLab
AIcrowd Submission Received #151632 - v5.5 (#2) · Issues · HaoheLiu / music-demixing-challenge-starter-kit · GitLab

#9

Hi @HaoheLiu,

I have added a new update which should help with this error, and we are monitoring newly failed submissions.

Meanwhile, also bumped the failed submission limit to 20, so the failed submissions count doesn’t stop you from making challenge-end submissions.

3 Likes