How to upload large files (size) to your submission

Managing Large Files in GitLab Repositories

When working with GitLab repositories, there is a limitation on the maximum file size for individual files, which is set to 50MB. Attempting to upload files larger than this limit will result in a failure during the git push operation, with the following error message:

remote: fatal: pack exceeds maximum allowed size
error: remote unpack failed: index-pack abnormal exit

To accommodate larger files, such as trained models or other large datasets, it is necessary to use Git Large File Storage (Git LFS). This document outlines the steps for using Git LFS, including how to migrate existing large files into Git LFS using git-lfs-migrate.

Setting Up Git LFS

Before you can use Git LFS, you must install it and set it up in your repository. Follow these steps to get started:

  1. Install Git LFS:

    • Execute the command git lfs install to install Git LFS on your system. This will update your Git hooks and initialize Git LFS.
    ❯ git lfs install
    Updated git hooks.
    Git LFS initialized.
    
  2. Track Large Files:

    • Use the git lfs track command to specify which file patterns to track with Git LFS. Replace "*.mymodel" with your file pattern.
    ❯ git lfs track "*.mymodel"
    Tracking "*.mymodel"
    
  3. Commit Changes:

    • Add the .gitattributes file and your large files to the staging area, and then commit them to your repository.
    ❯ git add .gitattributes
    ❯ git add some.mymodel
    ❯ git commit -m "Add large file support with Git LFS"
    ❯ git push origin master
    

Additional Notes

  • Windows Users: If you prefer not to use the terminal, the “Git for Windows” application is a user-friendly alternative.

  • Troubleshooting: If you encounter the error git: 'lfs' is not a git command. See 'git --help'., your Git version might be outdated and not include Git LFS. Install Git LFS using apt-get install git-lfs or brew install git-lfs, depending on your OS.

  • Migrating Existing Large Files: If large files have already been committed to your repository, and you’re unable to push changes due to the size limitation, use the git lfs migrate command to amend your Git history and migrate the large files to Git LFS. This process may require a force push to complete.

Resources

This documentation aims to provide a clear and concise overview of managing large files in GitLab repositories using Git LFS. For further assistance, please consult the linked resources or the Git LFS documentation.

6 Likes