Product Matching: Inference failed

Visual Product Recognition Challenge 2023
#24

@dipam, could you please check
#212567
#212566
It is failed in step “Build Packages And Env”, however I have changed only NN params.
As to me it is very strange…

1 Like
#25

@dipam Can you help me why my submission failed, please? AIcrowd

#26

@dipam Could you check submission: 213161?

The diagram shows that everything worked fine, but the status is failed:

Screenshot 2023-03-29 at 08.37.02
Screenshot 2023-03-29 at 08.37.021798×438 45.1 KB

#27

@dipam sorry to bother you again. but can you please let me know why 214218 and 212720 failed?

#28

@youssef_nader3 Both of these timed out

#29

@dipam , have you changed some settings of the server for inference?
Previously I have faced near 1 failed submitting per day, just rerunning helps.
however today I have changed only NN weights files and thats all, 1 get 1 submit ok, and 4 other weights with same size, same model all same just other epochs - failed.
As to me it is very strange…
Could you pls check it?

#214502

#214483

#214469

If it is timeout, how it could be if other weights are ok, or just resubmitting sometimes helps?

#30

@Mykola_Lavreniuk

All these timed out, they’re just barely above the time limit. The variation in runtime can be due to the slight variation in CPU type that the AWS nodes we provision can have. Hence resubmitting can sometimes help, but for consistency I suggest trying to bring down the compute time.

I understand that every second might matter, however the organizer has deemed 10 minutes to be a generous time limit for the kind of solutions they are looking for, hence the constraint.

#31

@dipam, thank you for the quick response

#33

@dipam Hi, can you please tell me why my submission # #215436 failed?

#34

@v_mudryi

I’ve added the error on your gitlab issue page for the submission.

1 Like
#35

@dipam Hello! Help me, please! Why my submission is failed? AIcrowd

#36

@dipam I checkout on successful submission and he also failed AIcrowd Please, help me!

#37

Hi, you can check the explanation on this message above. Hope it helps.

#38

Hi @dipam can you please tell me why my submission # # 216553 failed? I now that its late for submission, but just interested if it’s due to a timeout or something else.

#39

Hello, @dipam
I’m also curious what happened to sumbissions #216545 and #216531
Could you tell please?

#40

@ilia_denisov 216545 timed out, 216531 had KeyError: ‘product_id’

1 Like
#41

@v_mudryi 216553 timed out

1 Like
#42

@dipam Wondering why my submission failed on private set, but worked ok on public? Timed out on private set?
#216542

#43

Timeout has been relaxed by a certain amount, however we’ll likely remove the timeout for the private set as the model has already demonstrated the speed.

However, some other submissions are failing due to exceeding RAM or GPU OOM, which is a tricky situation and we’re yet to decide what to do about it.

This is not specific to your submission, if your submission failed due to timeout on private rest assured it will be retried.

3 Likes
#44

Hi @dipam,

Unfortunately, my submissions failed even after the retry run :frowning:

Submissions 216117 and 215394 failed during the Evaluate Score step, with the following error message:

Evaluate Scores: ValueError: Object arrays cannot be loaded when allow_pickle=False.

I suspect this might be a numpy.load error, as suggested here: python - ValueError: Object arrays cannot be loaded when allow_pickle=False - Stack Overflow. However, I’m not entirely sure.

In addition, my other submission (216096) failed due to a time-out error, and I thought the time-out will be removed for the private runs.

I’m wondering if there will be another re-run for the failed submissions.

Thank you.