Submission Errors

@nilabha: Had posted earlier on the relevant issue. The problem was the run.sh does not have execution permissions, so a chmod +x run.sh (and a subsequent commit) should fix the problem.

@mohanty
run.sh it seems already has executable permission, hence there is no change in my local files on doing chmod +x run.sh
permissions are -rwxrwxrwx in my local
I have looked at the permissions for the other files and they are the same.
Do you think there is some other error?

Thanks,
Nilabha

I pulled down the image built from your submission, and the permissions for run.sh indeed do not have any execution permission :

root@408dfe6f4a7c:~# ls -al
total 112
drwxr-xr-x 1 aicrowd aicrowd 4096 Jul 24 16:54 .
drwxr-xr-x 1 root    root    4096 Jun 14 08:09 ..
-rw-rw-r-- 1 aicrowd aicrowd 3033 Jul 24 16:46 aicrowd_helpers.py
-rw-rw-r-- 1 aicrowd aicrowd  194 Jul 24 16:46 aicrowd.json
-rw-r--r-- 1 aicrowd aicrowd  220 Jun 14 08:09 .bash_logout
-rw-r--r-- 1 aicrowd aicrowd 3771 Jun 14 08:09 .bashrc
-rw-rw-r-- 1 aicrowd aicrowd  339 Jul 24 16:46 build.sh
drwx------ 3 aicrowd aicrowd 4096 Jul 24 16:54 .cache
drwxrwsr-x 2 aicrowd aicrowd 4096 Jul 24 16:53 .conda
drwx------ 3 aicrowd aicrowd 4096 Jun 19 08:45 .config
drwxrwxr-x 1 aicrowd aicrowd 4096 Jul 24 16:46 data
-rw-rw-r-- 1 aicrowd aicrowd  349 Jul 24 16:46 debug.sh
drwxr-xr-x 2 aicrowd aicrowd 4096 Jul 24 16:53 .empty
-rw-rw-r-- 1 aicrowd aicrowd 2123 Jul 24 16:46 environment.yml
-rw-rw-r-- 1 aicrowd aicrowd   94 Jul 24 16:46 environ.sh
-rw-rw-r-- 1 aicrowd aicrowd   94 Jul 24 16:46 .gitattributes
-rw-rw-r-- 1 aicrowd aicrowd   57 Jul 24 16:46 .gitignore
drwxrwxr-x 1 aicrowd aicrowd 4096 Jul 24 16:46 models
-rw-r--r-- 1 aicrowd aicrowd  807 Jun 14 08:09 .profile
-rw-rw-r-- 1 aicrowd aicrowd 8559 Jul 24 16:46 README.md
-rw-rw-r-- 1 aicrowd aicrowd 8461 Jul 24 16:46 run.py
-rw-rw-r-- 1 aicrowd aicrowd   28 Jul 24 16:46 run.sh
drwxrwxr-x 1 aicrowd aicrowd 4096 Jul 24 16:46 sample
-rw-rw-r-- 1 aicrowd aicrowd  132 Jul 24 16:46 trainnew.csv

@nilabha: I fixed that in this commit : https://gitlab.aicrowd.com/nilabha/snake-species-identification-challenge/commit/c51854e038a5bb600f536bcafcd944735955def1

Best of luck

Hi @mohanty I am getting the following error in my submission:

RuntimeError: DataLoader worker (pid(s) 16) exited unexpectedly

(I read the following could fix the issue --shm-size 50G and runtime --runtime=nvidia could we try? do not know where to add it for the container config)

@mohanty
I have redone with a fresh copy from repo and updating the files.
Sorry for asking you again but I still get the error in my latest submission http://gitlab.aicrowd.com/nilabha/snake-species-identification-challenge/issues/22
Do you know what is the error?

Thanks,
Nilabha

I am feeling guilty having to ask you everytime :stuck_out_tongue:
I wouldn’t mind if all my logs are made public to be honest.

But on another note would the deadlines of this competition be extended. Due to less time left in the competition, I haven’t done much to replicate this submission process locally (I think there are some instructions in the link https://github.com/stanfordnmbl/neurips2019-learning-to-move-starter-kit). As of now I just do bash run.sh to test that the code is working in my local.

Looks like this is because the Dataloader is trying to spawn way too many workers. Can you set num_workers=0 in the DataLoader so that it does all the data loading in the main thread.

@mohanty
I have set num_workers=0
I still get the error.
https://gitlab.aicrowd.com/nilabha/snake-species-identification-challenge/issues/24
Could you share the error logs?

Thanks,
Nilabha

@nilabha Commented on the issue. Hope this helps!

@mohanty
I have been using trial and error to see which line of code the submission is failing, unfortunately I have crossed the limit of submission
Could you provide the error logs to debug the issue?
I have isolated the issue to few lines of code but it would be helpful to know what is the error.

Thanks,
Nilabha

@mohanty @arjun_nemani

Can you please help with the error logs. I keep getting an evaluation failure.
https://gitlab.aicrowd.com/nilabha/snake-species-identification-challenge/issues/31

I have tried reducing the batch size also in case of any memory issue. However the error still seems to be there.
From trial and error I notice that the error happens in the code line learn.tta function from the fast ai library.

Thanks,
Nilabha

@nilabha: Some images in the test set are corrupt unfortunately. And we have not removed them to stay consistent. But if you cannot read an image with PIL, then please use a random prediction for the same.

You can use the following way to solve this issue:
verify_images(test_images_path,delete=True) # test_images_path in run.py
This will fetch you 17686 files
You will have to have some logic to generate predicitions for corrupted files
I do it this way (not the best way):

  1. Use the sample submission file and replace the random given probabilities with predicted probabilities

Do test it locally. TTA takes a long time to complete (~7hrs)

@mohanty
I have put a workaround to find the images which fail loading and delete them and later add these probability which are all equal to 1/45
However I still get an error
https://gitlab.aicrowd.com/nilabha/snake-species-identification-challenge/issues/32

Can you please help with the error?

Thanks,
Nilabha

The error is :

Traceback (most recent call last):
  File "run.py", line 258, in <module>
    run()
  File "run.py", line 146, in run
    os.remove(sRemove)
OSError: [Errno 30] Read-only file system: '/test_data/2e23ade63c4e32b728a423ff19e52ef1.jpg'

Your code is trying to remove images from the test set here

Instead of trying to delete the corrupt files, please try to just append a random prediction to the final prediction CSV file.

@mohanty Evaluator is not picking up any of my submissions. Please look into it. Latest commit : https://gitlab.aicrowd.com/gokuleloop/snake-breed-identification/commit/c2d0dd9f55cd89796de929328b9a6e746941e774

1 Like

As mentioned in the starter kit, the tag names have to begin with “submission-“ , so pushing a tag called as say : submission-v010
should do the trick.

This was a loosely held rule and has only been enforced recently.

1 Like

You have only read access for the initial test images directory, so you can’t remove files from there. However, you can copy the whole test images directory to a local directory and run verify images there.

os.system("cp -r {0} test/".format(test_images_path))
verify_images("test", delete=True)

1 Like

Thanks kongas
I used the below code to remove the images

filter_func = lambda x: str(x) not in lsRemove
test_img = (ImageList.from_folder(path).filter_by_func(filter_func))

Though there is another error…
https://gitlab.aicrowd.com/nilabha/snake-species-identification-challenge/issues/33

@mohanty
Has the competition ended or will it restart again. Would have liked to get a score as my validation results were good.
Is it possible to get the error logs?