Best Way to Set Up Remote Git with Large Files

user_6491230 0
Viewed 128

I have a directory containing my code and very large files (e.g. data files and large deep learning models). Ignoring those large files, I have uploaded the code onto a GitHub repository.

Now, I also have set up a remote server (AWS EC2) where I have cloned that very repository. However, because I want to do work with the data and these large files, what's the best way to set up a good workflow? Should I upload those files to my server and just set up a .gitignore — is there a smarter way of doing this?

Sorry for the noob question, thanks!

remote-access github git
2 Answers

You're right not to store large data files in your repository. Data files aren't a good fit for Git repositories, even with Git LFS, because they're something that your code operates on, not your code itself.

If you're ignoring those paths on your workstation via .gitignore, then they should also be ignored on your server when you check them out in the same paths. This is the recommended for anything you don't intend to check into your repository, so if you haven't, it's a good idea.

As for your proposed solution, it seems completely reasonable: with the files ignored, you don't have to worry about checking them in accidentally, and you can move them wherever you need. If you need a way to transport incremental updates to the server, you could try rsync, which will work over SSH and can avoid transferring most of the data if the files are mostly the same.

user_87054320

Check this git-lfs https://git-lfs.github.com/ It may be very helpful for your case.

An open source Git extension for versioning large files

So as it says, if you want to versioning your large files (I guess yes for the learning models) or to make this files accessable from your git repo then use the above approach.

The other approach is as you said. Upload the files to the server and .gitignore them or even place them outside the repo.

user_114501740
Related