Today it took 10+ hours to receive the result of two submissions that we made. After submitting, we were watching the progress at AIcrowd GitLab issues. The submissions were successfully predicting questions. Then the updates froze and the submissions were labeled “evaluation_initated” for the next 8 hours. After 8 hours we saw that the submissions were labeled “failed”.
The debug logs do not show any errors. This is the same code that worked before (on a different track) and/or made more progress before (on same track).
Is it possible that the errors and/or inconsistent behavior is caused by AIcrowd servers? Should we just submit the same code again?
Has any team submitted the exact same code twice. And one time the code failed and one time the code was successful?
I just submitted the exact same code to the exact same track that failed earlier today. This time it succeeded. Does this mean that we need to repeatedly submit our failed code to AIcrowd server?
What is causing the inconsistent behavior of AIcrowd server? Do fails that occur because of AIcrowd server count toward are weekly track 1-4 limit of 20 and weekly track 5 limit of 3?
Hi. Can an admin please solve the following problem and provide a score for our last submission?
We just submitted our other code that failed yesterday and this time the AIcrowd GitLab issue says the error is caused by AIcrowd server. The server could not load the sentence-transformers/all-MiniLM-L6-v2 to evaluate the score. Below is copy and paste from reason for failure.
Evaluate Scores: OSError: We couldn’t connect to ‘https://huggingface.co’ to load this file, couldn’t find it in the cached files and it looks like sentence-transformers/all-MiniLM-L6-v2 is not the path to a directory containing a file named config.json. Checkout your internet connection or see how to run the library in offline mode at ‘https://huggingface.co/docs/transformers/installation#offline-mode’.
Our code successfully predicted all questions within all time limits and then AIcrowd server failed to load sentence transformer to compute our LB score
@Chris_Deotte Hi, we are now aware of the issue and are looking into it.
1 Like
@Chris_Deotte We have fixed the issue. Your submission has been re-initiated.
1 Like