Broken Pipeline error

Flatland
#1

I am getting a [32] broken pipeline error when submitting. Is anyone else facing this issue?

#2

Same here. The submission fails after this message, “Submission failed : [Errno 32] Broken pipe”.

#3

Hello @harshadkhadilkar @ankushojha

This was an issue on our end and is fixed now. You should be able to submit your solutions.

1 Like
#4

Just tried it, same thing happened.

https://gitlab.aicrowd.com/harshadkhadilkar/tracktricks/issues/99

#5

We started the evaluation for this commit and the future submissions should work fine. Please let us know if you still face any issues.

#6

Thanks @jyotish. However, I am now out of submission slots for today - it looks like the failed submissions got counted?

#7

@harshadkhadilkar, or anyone else who was affected from this outage: DM me the URLs of all your submissions that failed in this way and we will remove them from your count.

1 Like
#8

Looks like the problem still occurs. We’re suspending submissions until this is fixed for good!

Submission failed : [Errno 32] Broken pipe
#9

Hi all, the issue is now resolved.
In case your submission was counted twice or failed (not requeued yet) during the downtime, please let us know in below post.

New submissions are now being accepted.

Happy solving !

#10

Hi @jyotish, @MasterScrat, @shivam: I’m not sure if this is a related issue, but all my submissions seem to be failing since this morning. They move from image creation to “evaluation pending”, and then directly to “failed” without starting the evaluation. Looking at the Submissions page, a lot of other people may also be facing the same issue.

In case this is a technical fault, please re-queue my Saturday submissions. Thank you!

1 Like
#11

Update: A debug mode submission showed an error in the load model statement. This could be a keras/tensorflow version mismatch - was the standard docker image changed recently?

#12

@shivam do you know if a new version of the base image or repo2docker was deployed?

The other errors in the submission page are unrelated.

#13

I think this is the issue (see link below). A new version of the h5py library was released 22 hours ago as of now. This library is used to load/save keras models, among other things. Will try with a specific version requirement in the yml file.

2 Likes
#14

Nice catch! looks like this fixes it

#15

Yes, worked in debug mode just now. Ran out of my normal 5 slots.

In case anyone else runs into this issue: please ask them to enforce h5py==2.10.0 in the yml file.