Is there a max repo size?

Unity Obstacle Tower Challenge
#1

Hey, I was able to submit my build for evaluation, however my model weights are pretty large(4.4GBs) and it seems the build server is out of space? Here is the error I’m getting…

Step 27/33 : COPY src/ ${HOME}
failed to copy files: failed to copy directory: Error processing tar file(exit status 1): write /checkpoints/$store$_observation_ckpt.100.gz: no space left on device

Is there a max repo size or a recommend way to add large files to our docker image?

#2

https://docs.gitlab.com/ee/user/admin_area/settings/account_and_limit_settings.html

#3

So my repo size is 4.4GB under the 10GB limit stated there, my code is in gitlab so the problem is not there.

The problem seems to be in the build process in the repo2docker phase. The script works fine on my local box, but there seems to be some file size issue on their build server. Would like some guidance on if there is away to remedy that issue.

#4

Thanks for reporting this @banjtheman, investigating this.

#5

@banjtheman do you mind linking the submission that failed for you? I found @ChenKuanSun’s but not yours.

#6

https://gitlab.aicrowd.com/banjtheman/obstacle-tower-challenge/issues/2

#7

The issue has been fixed, and you can retry the submissions now.
We were having docker image size limitation (at intermediate stages of docker build as well) of 10G which is now increased to 20G.

#8

Great the image was able to build successfully now

#9

Hi there,

Today, I push 2 commit into gitlab and both failed at the steps during image_building:

Step 27/33 : COPY src/ ${HOME}
failed to create rwlayer: devmapper: Thin Pool has 127713 free data blocks which is less than minimum required 163840 free data blocks. Create more free space in thin pool or use dm.min_free_space option to change behavior

I’ve checked my repo and it’s about 270 MB in size.

#10

Ha yup same issue just now, looks like the server’s docker volume might be full

#11

Ok, seems like aicrowd server-side issue then.

#12

I did a quick pruning, your submissions should go through now.
But boy !! Some of the submitted images are 22GB huge -_- !!!

@shivam : Sending you what I did via DM for reference.

#13

Hi mohanty,
seems the server’s docker volume is full again.

Step 27/33 : COPY src/ ${HOME}
failed to create rwlayer: devmapper: Thin Pool has 126635 free data blocks which is less than minimum required 163840 free data blocks. Create more free space in thin pool or use dm.min_free_space option to change behavior

#14

Can you please try again now ?

#15

Thanks for that, My submission was successfully built.
But in the evaluation part, my code get killed by some reason.

/home/aicrowd/run.sh: line 10: 14 Killed python run.py

I guess it occurs when my agent load data, which would consume around 6GB memory(tested on my machine)

https://gitlab.aicrowd.com/andwetry/obstacle3/issues/15

#16

Seeing similar issue, suspect its the same thing happens right when agent tries to load checkpoints

https://gitlab.aicrowd.com/banjtheman/obstacle-tower-challenge/issues/12

#17

I acknowledge this issue.
I am increasing the available memory for the agents containers to 10GB.
Can you please try resubmitting again ?

#18

@banjtheman: I managed to get your submission running on the server just by increasing the available resources. Your submission should hopefully work now. Best of luck.

Also please refer to : [Admin] Good News for Dopamine Lovers!
For some inputs on saving checkpoints in the repository to ensure that the images are not too bloated !

#19

@mohanty
seems the server’s docker volume is full again.

Step 27/33 : COPY src/ ${HOME}
failed to create rwlayer: devmapper: Thin Pool has 161056 free data blocks which is less than minimum required 163840 free data blocks. Create more free space in thin pool or use dm.min_free_space option to change behavior
#20

This issue should now be resolved. Please feel free to reach out if you see this again.