Managing large binary files with Git

pi_15274 0
Viewed 200734

I am looking for opinions of how to handle large binary files on which my source code (web application) is dependent. We are currently discussing several alternatives:

  1. Copy the binary files by hand.
    • Pro: Not sure.
    • Contra: I am strongly against this, as it increases the likelihood of errors when setting up a new site/migrating the old one. Builds up another hurdle to take.
  2. Manage them all with Git.
    • Pro: Removes the possibility to 'forget' to copy a important file
    • Contra: Bloats the repository and decreases flexibility to manage the code-base and checkouts, clones, etc. will take quite a while.
  3. Separate repositories.
    • Pro: Checking out/cloning the source code is fast as ever, and the images are properly archived in their own repository.
    • Contra: Removes the simpleness of having the one and only Git repository on the project. It surely introduces some other things I haven't thought about.

What are your experiences/thoughts regarding this?

Also: Does anybody have experience with multiple Git repositories and managing them in one project?

The files are images for a program which generates PDFs with those files in it. The files will not change very often (as in years), but they are very relevant to a program. The program will not work without the files.

large-files version-control binaryfiles git
13 Answers

If the program won't work without the files it seems like splitting them into a separate repo is a bad idea. We have large test suites that we break into a separate repo but those are truly "auxiliary" files.

However, you may be able to manage the files in a separate repo and then use git-submodule to pull them into your project in a sane way. So, you'd still have the full history of all your source but, as I understand it, you'd only have the one relevant revision of your images submodule. The git-submodule facility should help you keep the correct version of the code in line with the correct version of the images.

Here's a good introduction to submodules from Git Book.

Pat_Notz_8250

I would use submodules (as Pat Notz) or two distinct repositories. If you modify your binary files too often, then I would try to minimize the impact of the huge repository cleaning the history:

I had a very similar problem several months ago: ~21 GB of MP3 files, unclassified (bad names, bad id3's, don't know if I like that MP3 file or not...), and replicated on three computers.

I used an external hard disk drive with the main Git repository, and I cloned it into each computer. Then, I started to classify them in the habitual way (pushing, pulling, merging... deleting and renaming many times).

At the end, I had only ~6 GB of MP3 files and ~83 GB in the .git directory. I used git-write-tree and git-commit-tree to create a new commit, without commit ancestors, and started a new branch pointing to that commit. The "git log" for that branch only showed one commit.

Then, I deleted the old branch, kept only the new branch, deleted the ref-logs, and run "git prune": after that, my .git folders weighted only ~6 GB...

You could "purge" the huge repository from time to time in the same way: Your "git clone"'s will be faster.

Vlookward_161350

In my opinion, if you're likely to often modify those large files, or if you intend to make a lot of git clone or git checkout, then you should seriously consider using another Git repository (or maybe another way to access those files).

But if you work like we do, and if your binary files are not often modified, then the first clone/checkout will be long, but after that it should be as fast as you want (considering your users keep using the first cloned repository they had).

user_618380

git clone --filter from Git 2.19 + shallow clones

This new option might eventually become the final solution to the binary file problem, if the Git and GitHub devs and make it user friendly enough (which they arguably still haven't achieved for submodules for example).

It allows to actually only fetch files and directories that you want for the server, and was introduced together with a remote protocol extension.

With this, we could first do a shallow clone, and then automate which blobs to fetch with the build system for each type of build.

There is even already a --filter=blob:limit<size> which allows limiting the maximum blob size to fetch.

I have provided a minimal detailed example of how the feature looks like at: How do I clone a subdirectory only of a Git repository?

user_8952450

Git LFS is the answer

# Init LFS
git lfs install
git lfs track "large_file_pattern"

# Then follow regular git workflow
git add large_file
git commit -m "Init a very large file"
git push origin HEAD

Behind the scene, git lfs would create a reference to your large file and does not store directly in the git repo

For more info: https://git-lfs.github.com/

user_38318450
Related