I'm looking for a way to give users access to data of a specific version of a repo through raw.githubusercontent.com, without massively increasing the size of the already very large repo.
Context: I have a GitHub repo that contains a large amount of data created by a text reuse algorithm. The same algorithm is run twice a year on a corpus of texts that undergoes change, so the data in this repo changes twice a year.
In order to allow access to a specific version of the text reuse data, I was thinking of using tags; in my understanding, a tag is simply a pointer to a specific commit, so it should not significantly increase the size of the repo.
However, this does not seem to be the case according to an experiment I ran (see below): when I push a tag to GitHub, it also creates a zip file that contains the full data at the time of the commit. It even does this when I have not pushed that commit to GitHub yet, and even if I use the lightweight tag format. It seems that this will increase the size of my GitHub repo enormously.
Is there a way to add tags without the creation of the .zip file?
If not, I was thinking of creating a new branch each time the algorithm is run; the branch would not be updated but only serve as a pointer to that specific commit. Would this increase the size of the repo? Are there other downsides of using a branch for this?
Experiment:
Setting up the experiment:
- Create a test repo and clone it
- create a file in it, with the content "DUMMY DATA – V1"
- add and commit
Step 1: annotated tag
- Create an annotated tag:
git tag -a v1.0 -m "First version with dummy data" - Push the repo to GitHub:
git push origin main - This has pushed only the new file to the repo, not the tag; the tag has to be pushed separately:
git push origin v1.0 - Now the tag appears on GitHub; it is associated with a zip file that contains the file with content "DUMMY DATA – V1"
- Check whether you can access the file using raw.githubusercontent (it works, after a couple of minutes): https://raw.githubusercontent.com////
Step 2: annotated tag, push tag first
- Make changes to the file: change the content to "DUMMY DATA – V2"
- Add and commit the file
- Add a new annotated tag:
git tag -a v2.0 -m "Second version with new dummy data" - This time, we'll push the tag before we push the commit:
git push origin v2.0 - The new tag appears on GitHub, and it is associated with a .zip file that contains the file with contents "DUMMY DATA – V2"; in the repo on GitHub, the file still has the original content "DUMMY DATA – V1" => The zip file is thus not created by GitHub on the server but during the process of pushing the tag!
- push the commit:
git push origin main
Step 3: lightweight tags
- Make changes to the file: change the content to "DUMMY DATA – V3"
- Add and commit the file
- Create a so-called lightweight tag (omit the "-a" flag, and the message):
git tag v3.0 - Again, we push the tag before we push the commit:
git push origin v3.0 - The same thing happens as in step 2: the new tag appears on GitHub, and it is associated with a .zip file that contains the file with contents "DUMMY DATA – V3"; in the repo on GitHub, the file still has the content "DUMMY DATA – V2" => lightweight tags are not so lightweight: they also create a zip file!