Dirty trick to keep commit hashes when rewriting Git history?

user_21728 0
Viewed 753

Note: there's a similar question How to keep commit hashs not change when use git filter-repo rewrite the history but the answers focus on how Git cannot do that. In this question, I'd like to explore whether, in theory, it's possible to write a custom script to keep the commit hashes.

Git filter-branch and BFG Repo-Cleaner are two popular tools to remove large files and other things from a repo history. They lead to different commit SHAs / hashes, which is how Git works as it "fingerprints" the contents of the commit, its parents etc.

However, we're in a situation where the unfortunate large file commit happened a while ago and we have all sorts of references to newer commits e.g. in GitHub issues ("see commit f0ec467") and other external systems. If we used filter-branch or BFG, lot's of things would break.

So I came here to ask whether there's some dirty, low-level trick how to keep commit IDs / SHA-1 even for rewritten commits. I imagine that for a bad commit that we want to rewrite, a custom script would create a new Git object but "hardcoded" the same / old SHA-1, skipping the calculation of it. The newer commits (its children / descendants) should continue working I think (?!).

If this couldn't work, I'd like to understand why. Does Git check that hashes agree with the actual contents regularly? Does it do so only during some operations, like gc or push or pull?

(I know this is a very thin ice, I'm just technically exploring our options before we accept that we'll have a large binary in our repo forever, with all the implications like having much larger backups forever, full clones taking longer, etc.)

UPDATE: There's now an accepted answer but at the same time, no answer mentions git replace which might be the solution to this? I've done some basic experiments but am not sure yet.

git
2 Answers

I included a link as a comment, but in fact, breaking SHA-1 doesn't help very much.

The problem is that Gits exchange objects by comparing object hash IDs. These are currently SHA-1 (see the other question and its answer for some future possibilities). If you manage to break SHA-1, and produce a new input object that generates the same hash ID, you could:

  • rip the old object out of your Git's object database, then
  • insert the new object into your Git's database

and from then on, your Git would see only the new object, instead of the old one. But when you connect your Git to some other Git, and your Git says to that other Git: I have object a123456..., would you like it? the other Git might just answer: No thanks, I already have that one. They have the old one, of course. So you've made your Git incompatible with their Git, but gained nothing from this.

If the other Git doesn't have the object in question, well, then you're OK! They will ask for your copy and you can hand that over.

Commit and tag objects have room in them for somewhat-arbitrary (not completely arbitrary) user data. This is where you would put your perturbable data for breaking SHA-1. Tree objects are less friendly, but as long as you can do what you need to with commit and tag objects, you can probably bypass this.

As for where to get the compute power, well, the price of a large group of Raspberry Pi computers is coming down....

Edit: I forgot to address this question:

Does Git check that hashes agree with the actual contents regularly?

Yes. In fact, it does this check every time it extracts an object by its hash ID. Remember that the bulk of most repositories is the object database, which is a simple key-value store. The key is the hash ID and the data stored under that key represent the object. Git uses the key to do the lookup, then verifies that the stored data hash to that key, to make sure the stored data were not corrupted by a disk or memory error.

user_12564520

A commit ID incorporates the commit IDs of its parents. This means if two commits have the same ID Git knows not just that the two commits are equal, but also that their entire histories are equal. This is fundamental to how Git works, particularly push and pull. Mess with it at your peril.

It's possible you could do something clever with git-replace, but I have no experience with that.

If this couldn't work, I'd like to understand why. Does Git check that hashes agree with the actual contents regularly? Does it do so only during some operations, like gc or push or pull?

git gc may have issues, but git fsck would lose its mind. You'd never be able to repair a broken repository. And as torek says, pushes and pulls between old and new repositories will get very confused.

I would recommend instead keeping an copy of the original repository around to reference. When you find a reference to an old ID you can still look it up. And if you judiciously rewrite them to reference the equivalent commit in the new repository, eventually you won't need the old repo anymore.

You could speed up this process by searching for hexadecimal strings, checking if they match a commit ID, and replacing it with the new commit ID. A mapping of old to new can be obtained by running git log --pretty='format:%H' on both repositories and comparing them one-to-one.

Update

If you really, really need those Github links to work, you could write an http proxy which redirects https://github.com/your-org/your-repo/commit/oldcommitid to https://github.com/your-org/your-repo/commit/newcommitid.

user_146600
Related