How to upload large files (size) to your submission

shivam · November 21, 2019, 5:07am

Managing Large Files in GitLab Repositories

When working with GitLab repositories, there is a limitation on the maximum file size for individual files, which is set to 50MB. Attempting to upload files larger than this limit will result in a failure during the git push operation, with the following error message:

remote: fatal: pack exceeds maximum allowed size
error: remote unpack failed: index-pack abnormal exit

To accommodate larger files, such as trained models or other large datasets, it is necessary to use Git Large File Storage (Git LFS). This document outlines the steps for using Git LFS, including how to migrate existing large files into Git LFS using git-lfs-migrate.

Setting Up Git LFS

Before you can use Git LFS, you must install it and set it up in your repository. Follow these steps to get started:

Install Git LFS:
- Execute the command git lfs install to install Git LFS on your system. This will update your Git hooks and initialize Git LFS.
```
❯ git lfs install
Updated git hooks.
Git LFS initialized.
```
Track Large Files:
- Use the git lfs track command to specify which file patterns to track with Git LFS. Replace "*.mymodel" with your file pattern.
```
❯ git lfs track "*.mymodel"
Tracking "*.mymodel"
```

Commit Changes:

Add the .gitattributes file and your large files to the staging area, and then commit them to your repository.

❯ git add .gitattributes
❯ git add some.mymodel
❯ git commit -m "Add large file support with Git LFS"
❯ git push origin master

Additional Notes

Windows Users: If you prefer not to use the terminal, the “Git for Windows” application is a user-friendly alternative.
Troubleshooting: If you encounter the error git: 'lfs' is not a git command. See 'git --help'., your Git version might be outdated and not include Git LFS. Install Git LFS using apt-get install git-lfs or brew install git-lfs, depending on your OS.
Migrating Existing Large Files: If large files have already been committed to your repository, and you’re unable to push changes due to the size limitation, use the git lfs migrate command to amend your Git history and migrate the large files to Git LFS. This process may require a force push to complete.
- Further guidance on migrating existing repository data to LFS can be found in the Git LFS Tutorial: Migrating existing repository data to LFS.

Resources

For a comprehensive guide on why and how to use Git LFS, refer to this tutorial on DZone.
Detailed instructions on using git-lfs-migrate to migrate existing large files to Git LFS can be found in the Git LFS Migrate manual page.

This documentation aims to provide a clear and concise overview of managing large files in GitLab repositories using Git LFS. For further assistance, please consult the linked resources or the Git LFS documentation.