Getting error in siam-mot baseline submission "pack exceeded maximum allowed size"

I’m getting an error “remote: fatal: pack exceeds maximum allowed size” while pushing siam-mot baseline to the repo.

2 Likes

Hello @vishleshak

You need to use Git LFS to push the large files in your repository (your saved models, etc.,). You can find more information on how to use LFS here

I did that. List of the commands in order:

  1. git clone siam-mot starter-kit
  2. git remote add aicrowd git@gitlab.aicrowd.com:{aicrowd_username}/airborne-detection-starter-kit.git
  3. git lfs install
  4. git lfs track “*.model”
  5. git add .gitattributes
    6 git commit
  6. git push aicrowd master

I haven’t changed anything other than those mentioned in the siam-mot template.

1 Like

Hi @vishleshak,

The starter kit code already contains the model files in git-lfs etc, due to which you shouldn’t have face this issue. I assume you downloaded repository (not clone) and trying to recreate it?


The error you shared is happening because you have used example i.e. for *.model files directly.
While the SiamMOT baseline contains *.pth files.

You need to repeat the above commands for *.pth instead of *.model.

In case, you have already committed *.pth files without git-lfs in your repository, please use git lfs migrate command as well.

Yes, it should be *.pth. I made a mistake while posting the details. Started with a fresh clone, did everything as mentioned, still getting the same error.

1 Like

Hi @vishleshak,

I suspect that you did git commit before doing git lfs install, etc.

But we can debug it in case you are still facing the issue, can you share output for following commands to help in debugging?

❯ git ls-files | xargs ls -l | sort -nrk5 | head -n 10
❯ git ls-tree -r -l --abbrev --full-name HEAD | sort -n -r -k 4 | head -n 10
❯ git lfs ls-files
❯ git --no-pager log --pretty=format:"%h%x09%an%x09%ad%x09%s" | head -n 10

Thanks

As asked output of commands listed below

  1. git ls-files | xargs ls -l | sort -nrk5 | head -n 10
-rw-rw-r-- 1 pauli pauli 95833603 Jun  9 11:59 siam-mot/models/DLA-34-FPN_box_track_aot_d4.pth
-rwxrwxr-x 1 pauli pauli 63228658 Jun  9 11:51 siam-mot/.cache/torch/hub/checkpoints/dla34-ba72cf86.pth
-rw-rw-r-- 1 pauli pauli 21482565 Jun  9 11:51 data/dataset-playground.ipynb
-rw-rw-r-- 1 pauli pauli    54398 Jun  9 11:51 core/metrics/airborne_metrics/calculate_airborne_metrics.py
-rw-rw-r-- 1 pauli pauli    30891 Jun  9 11:51 core/metrics/airborne_metrics/match_groundtruth_results.py
-rw-rw-r-- 1 pauli pauli    19380 Jun  9 11:51 core/metrics/airborne_metrics/calculate_encounters.py
-rw-rw-r-- 1 pauli pauli    15273 Jun  9 11:51 siam-mot/siammot/modelling/backbone/dla.py
-rw-rw-r-- 1 pauli pauli    12428 Jun  9 11:51 siam-mot/siammot/modelling/track_head/EMM/target_sampler.py
-rw-rw-r-- 1 pauli pauli    11915 Jun  9 11:51 core/metrics/run_airborne_metrics.py
-rw-rw-r-- 1 pauli pauli     9288 Jun  9 11:51 siam-mot/siammot/modelling/track_head/track_utils.py
  1. git ls-tree -r -l --abbrev --full-name HEAD | sort -n -r -k 4 | head -n 10
100644 blob 12a23d3 21482565	data/dataset-playground.ipynb
100644 blob 2f9bd23   54398	core/metrics/airborne_metrics/calculate_airborne_metrics.py
100644 blob 8eeae72   30891	core/metrics/airborne_metrics/match_groundtruth_results.py
100644 blob d57cedc   19380	core/metrics/airborne_metrics/calculate_encounters.py
100644 blob e45031d   15273	siam-mot/siammot/modelling/backbone/dla.py
100644 blob 62aef13   12428	siam-mot/siammot/modelling/track_head/EMM/target_sampler.py
100644 blob 5730ccf   11915	core/metrics/run_airborne_metrics.py
100644 blob 3c0e1d9    9288	siam-mot/siammot/modelling/track_head/track_utils.py
100644 blob 1ae8bac    9230	docs/DATASET.md
100644 blob 806e8e6    8610	siam-mot/siammot/data/image_dataset.py
  1. git lfs ls-files
ba72cf8642 * siam-mot/.cache/torch/hub/checkpoints/dla34-ba72cf86.pth
34afba9240 * siam-mot/models/DLA-34-FPN_box_track_aot_d4.pth
  1. git --no-pager log --pretty=format:"%h%x09%an%x09%ad%x09%s" | head -n 10
ca08897	paulids	Wed Jun 9 12:12:25 2021 +0530	changed aicrowd.json
8b446cc	Shivam Khandelwal	Thu Jun 3 20:21:02 2021 +0000	Added note about hardware used
bf5277e	Shivam Khandelwal	Mon May 31 18:45:55 2021 +0000	SiamMot disqualification criteria added
7676913	Shivam Khandelwal	Fri May 28 10:26:51 2021 +0000	Removed duplicate content about SiaMOT
32967fa	Shivam Khandelwal	Fri May 28 09:03:22 2021 +0000	Rebased with working SiamMOT codebase
24acdc5	Shivam Khandelwal	Fri May 28 03:39:11 2021 +0000	Missed siammot/data/ due to gitignore
6c6db7d	Shivam Khandelwal	Fri May 28 02:51:01 2021 +0000	SiamMot cache directory for submissions
27b971d	Shivam Khandelwal	Fri May 28 02:42:40 2021 +0000	SiamMot orchestration
17822c0	Shivam Khandelwal	Fri May 28 01:47:34 2021 +0000	Added siam-mot implementation in starter kit
4ffd45c	Shivam Khandelwal	Sat May 22 00:30:35 2021 +0000	bbox format specified to remove confusion

Hi @vishleshak,

Everything looks perfectly fine in your repository based on the outputs.

Can you share the exact error on doing git push ideally with the full command you are doing so?
It is still giving you “remote: fatal: pack exceeds maximum allowed size”? :thinking:

  1. git push aicrowd master
    output
Locking support detected on remote "aicrowd". Consider enabling it with:
  $ git config lfs.https://gitlab.aicrowd.com/paulids/airborne-detection-starter-kit.git/info/lfs.locksverify true
Uploading LFS objects: 100% (2/2), 159 MB | 0 B/s, done.                                                                                                                                      
Enumerating objects: 341, done.
Counting objects: 100% (341/341), done.
Delta compression using up to 8 threads
Compressing objects: 100% (182/182), done.
remote: fatal: pack exceeds maximum allowed size
error: remote unpack failed: index-pack abnormal exit
To gitlab.aicrowd.com:paulids/airborne-detection-starter-kit.git
 ! [remote rejected] master -> master (unpacker error)
error: failed to push some refs to 'git@gitlab.aicrowd.com:paulids/airborne-detection-starter-kit.git'
1 Like

Hi @vishleshak,

Thanks for raising the issue and it is indeed an issue in default commands shared in the starter kit (happening because the total size of the repository has grown over time to be pushed at once).

You can push your repository using the below command as a workaround (only needed when pushing for the first time). We will meanwhile look for how to fix it cleanly as well.

❯ git rev-list --reverse master | ruby -ne 'i ||= 0; i += 1; puts $_ if i % 10 == 0' | xargs -I{} git push aicrowd +{}:refs/heads/master 

Thank you. I was able to push the repo.

1 Like

This issue has been resolved, and the workaround is no longer needed!

Why was it failing?

The final pack file was too large to be pushed. Shifting data/dataset-playground.ipynb to LFS brought back the pack file size within acceptable limits.

But I’m still getting the same error!

The starter kit has been updated, and in order for this to work, you’ll need to clone it again.

2 Likes

(possibly resolved)

===================
Docker image not found, building…
Error processing tar file(exit status 1): write /home/aicrowd/siam-mot/models/DLA-34-FPN_box_track_aot_d4.pth: no space left on deviceError processing tar file(exit status 1): write /home/aicrowd/siam-mot/models/DLA-34-FPN_box_track_aot_d4.pth: no space left on device

Hi @david_lander,

We had a storage issue in Docker builds yesterday but is resolved.
I missed re-queuing your submission and have done now. :raised_hands:

1 Like

(seemingly resolved using !git lfs migrate import --everything)

Any commits with models are showing this error:
remote: fatal: pack exceeds maximum allowed size

(and work when commenting out the shutil.copyfile line below)

Code below:
!ssh -Tvvv git@gitlab.aicrowd.com

!apt-get install git-lfs
!git clone http://gitlab.aicrowd.com/amazon-prime-air/airborne-detection-starter-kit.git
os.chdir(“airborne-detection-starter-kit”)
!git lfs fetch --all
!git lfs track *.pth *.ipynb *.ckpt
shutil.copyfile(SOURCE_PATH, ‘model.pth’)
!git remote add aicrowd git@gitlab.aicrowd.com:nq/test0821k7.git
!git lfs track *.pth *.ipynb *.ckpt
!git add .gitattributes
cd /content/airborne-detection-starter-kit
!git add .
!git commit -m ‘Commit’
!git push aicrowd master

1 Like