Binary data: How should I make it available if I do not commit them to Git?

Viewed 49

I am using Python and Git for scientific applications. I often develop software tools that consist of a relatively small codebase and some large binary datasets.

Ideally, I want the software to work "out of the box", without the user having to manually download additional binary files. If I commit the binary data to Git, users can install everything using a single pip command (pip install git+https://github.com/myuser/myrepo). However, committing binary files is considered bad practice for a number of good reasons, such as slow download times if users want to upgrade the codebase. Also, a small backwards-compatible change to the binary files may lead to a large increase in the repo size.

Alternatively, I can put the binary data on an external server and let the software download the data if not present on the system (or if a newer version is available). Is there a preferred way to implement this procedure? In particular, is there a standard "download location" where the datasets should be placed?

1 Answers

For big files, you can enable Github's LFS service.

Git Large File Storage (LFS) replaces large files such as audio samples, videos, datasets, and graphics with text pointers inside Git, while storing the file contents on a remote server like GitHub.com or GitHub Enterprise.

I think, if your data will not change very often, you can just put your data in github release as attachment.

Related