Common Mistakes and Tips

NLP Feature Engineering-9be8f9
#1

By looking a lot of submission getting failed, we tried to look into every issue and there were some common mistakes by participants while making the notebook submission -

  1. The markdown text such as Install packages, Define Preprocessing code, Prediction phase is used by our evaluation server to identify the part of notebook run as per the rules of the challenge. These should not be removed or altered.

  2. Evaluation timed out. View the submission for more details.
    This is due to your submission taking too much time to make predictions, make sure your code doesn’t take more than 15 minutes to make predictions.

  3. Inference failed. View the submission for more details.
    There can be many reasons for this error, but as far as we saw in this challenge, this error mainly comes up due to participants uploading too big submission files, which filled up memory completely in our evaluation systems and caused the error. While we have increased the memory on our systems, make sure your submission never goes above 500 MB.

  4. “FileNotFoundError: [Errno 2] No such file or directory: ‘submission/assets’”
    It’s because the submission.zip didn’t contain the assets folder in the submission.zip.

    And in many other cases, the zipped file uploaded contained another folder and inside that folder contained the actual submission files. Something like this -

submission.zip
└── submission
  ├── submission.csv
  └── original_notebook.ipynb

which causes the issue. Make sure you zip the directory in the correct way.

If you are using Colab, you probably won’t need to zip folder as aicrowd-cli takes care of that :slight_smile:

In case you face any other issue, please feel free to reach out to us and we will be happy to help :slight_smile:

2 Likes
#2
#3

@Shubhamaicrowd - My inference error mentions a package not being imported, while the code runs fine locally. Can you suggest what could went wrong?

#4

We are looking into the issue, will give you updates asap.

#5

Thanks for the response.

#6

This comp is coming to an end and I would like to thank you for all your work and efforts, but as off now, there are more failed submissions than graded ones :-/ As a feedback, I would suggest that you make the process in a comp like that extremely clear and with a very minimal notebook example.

The notebook provided for this comp has way more things than the minimal for people make submission and it was great for learning purposes, but for a “sample submission” purpose it was not ideal. Maybe make two the next time? One as simple as possible, showing just the requirements and instructions, and then one more detailed and more advanced?

Speaking for myself, as off now, I could only get one off many submissions to run properly ant it was not consistent. I tried the same code with a new model having only different weights and it failed. Also the log provided by the platform also does not help.

Again, great work and great comp. I think the Blitz idea great and will recommend to others.

Tks!

2 Likes
#7

Thanks a lot for the feedback :slightly_smiling_face:

The notebook provided for this comp has way more things than the minimal for people make submission and it was great for learning purposes, but for a “sample submission” purpose it was not ideal. Maybe make two the next time? One as simple as possible, showing just the requirements and instructions, and then one more detailed and more advanced?

I actually really liked the idea, we will make sure in the future to put a minimal required submission notebook in the challenge.

I tried the same code with a new model having only different weights and it failed. Also the log provided by the platform also does not help.

Yep, This is something that we are actively working on will make sure to have better & informative logs from the next blitz.

Cheers
Shubhamai

2 Likes