How to get commits from another repo, when mine lost history?

Viewed 103

There was an OriginalRepo, with a commit history like this:

A -- B -- C

MyRepo was originally a fork of it. However, at some point, MyRepo was migrated and all the commit hashes were changed. So it became something like this:

A# -- B# -- C#

Development continued on both repos independently, so now they look something like this:

OriginalRepo: A -- B -- C -- D

MyRepo: A# -- B# -- C# -- X

Question

What would be the best way to get "D" into MyRepo, without losing history or "X"? (assume that both "X" and "D" are actually a range of commits, and contain merge commits as well)

I tried git cherry-pick FirstCommitInD^..LastCommitInD -m 1 HEAD, but when it gets to a merge commit it seems to be getting all the changes from A to D (probably because of the changed commit hashes).

1 Answers

First—and this is important-the commit hashes did not change. The modified repository has all-new commits. The new commits are just different commits. The old commits still exist: those are the ones they have in their repository (and you will, at some point, have again in yours).

In general, Git identifies commits by their hash ID. In a fundamental sense, the hash ID is the commit. Two Gits decide whether they have the same commits by comparing just the hash IDs.

This is what makes history rewriting "bad". A new repository, with all-new commits, is no longer compatible with an old repository. The history in a repository is the set of commits in the repository. So your history and their history are now completely disjoint.

What would be the best way to get "D" into MyRepo, without losing history or "X"? (assume that both "X" and "D" are actually a range of commits, and contain merge commits as well)

To get commit D literally, you must have your Git call up their Git and have them list out their commit hash IDs for your Git, and then have your Git pick up their commit. When D is a range of commits, you have to pick up the entire range. The easiest way to do this, by far, is to just have your Git pick up every one of their commits, so that you have all of your commits plus all of their commits. Instead of:

OriginalRepo: A -- B -- C -- D

MyRepo: A# -- B# -- C# -- X

you will have:

A  -- B --  C  -- D   <-- originalrepo/main

A# -- B# -- C# -- X   <-- main

in your repo, for instance. You now have both sets of commits, so you have both histories. History is nothing but commits after all. Your branch name(s), such as main or master, find the last commit(s) in your repository. The commits themselves point backwards to earlier commits, so that X points back to C#, which points back to B#, and so on. Meanwhile, your remote-tracking names, such as originalrepo/main, point to the last commit(s) that you got from their repository (now in your repository too). This lets your Git find D easily; D points back to C, which points back to B, and so on.

No part of any existing commit can ever change. That's why, when rewriting history in your repository earlier, your Git had to make new commits, with new and different hash IDs. Their Git and your Git (and every Git in the world) all agree that your new, rewritten commits get those new hash IDs—this was true even before you did the rewrites1—and that the originals get the original commit hash IDs.

This git fetch trick is simple and easy. You have commit D in your repo, now. But that's not really what you meant and want, is it? You actually want commit D to somehow join up with your commit X. Well ... that's possible, but harder: you will have to decide how you'd like to have this come about, and what, if anything, you plan to do with all of this in the future.


1This is why Git hash IDs are so big and ugly: we have to pre-allocate every hash ID for every commit that will ever exist in the future, and reserve it to that commit. This is a sort of fools' errand, so we don't actually do it in advance, we just do it as the commit data and hash IDs show up. But commit hash IDs are determined algorithmically based on the content, using a cryptographic hash. The pigeonhole principle tells us that this concept will fail someday; the size and efficacy of the hash pushes that someday into the future. The bigger and better the hash function, the further off is the Day Of Reckoning, when Git stops working.

More precisely, Git doesn't exactly stop working (you can still use your own files just fine). It just stops talking to some other Git because they've both used the same hash ID for a different internal object. You can no longer fetch or push: we need a new hash function. That day has not yet arrived for SHA-1, and Git is already moving to SHA-256 to push the day further down the road again.


How to deal with rewritten histories

Because what Git cares about are commits, as identified by their hash IDs, you really have just a few options:

  • Undo the rewrite entirely
  • Maintain parallel but separate histories
  • Join the rewrite to the original

All three of these options are terrible, each in a different way. The "undo the rewrite" method will have the best outcome, but is the hardest to do, usually by far. The "maintain parallel but separate histories"—sort of a software apartheid—means lots of extra work in the future, but at least that work is relatively amortized: you just copy their D commit(s) to new D# commit(s) that add on to your history, and then, as they make new commits, you keep on copying them. The "join" method means running git merge with --allow-unrelated-histories,2 resolving conflicts—there are almost certainly going to be some, perhaps many—and then making a merge commit, or multiple merge commits. After that, though, you'll be able to use Git to merge new commits they make, relatively easily; the terrible-ness is mainly the up-front work (like the "undo history rewrite" version) plus the fact that now you're dragging around a duplicated early history.


2Allowing unrelated histories was the default and only option, until Git 2.9. It turned out that this was causing too many causal Git users too much trouble, I guess, so they changed it to be forbidden by default, with an option to allow it.


How to undo the rewrite

Given the relatively simple drawing, with each A-B-C-D representing a single commit, this would be pretty easy. You'd just match up your A#—what I usually call A'—to their A and so on. That leaves you with one single commit X whose parent is C#. To undo the rewrite, you just:

  • get all their commits (git fetch);
  • create a new branch name that selects existing commit C;
  • use git cherry-pick to copy your single commit X to a new commit X';
  • ditch your commits entirely, in favor of this new history that ends at X'; and
  • get every other Git repository that has X to switch to using X' instead of X, and with that, the A-B-C-X' sequence.

For a repository over on GitHub that you control, you'd need git push --force to accomplish the last step. Other users that have cloned that repository—including anyone who forked it, since forking is just a clone with features added—will then have to update their repositories as well, and switch to the "un-rewritten" history.

Since X is not just a single commit, you'll need multiple cherry-pick operations. The git rebase command is generally the one to use here, since it operates by doing repeated cherry-picking. However, any internal branch-and-merge operations require repeating the merges, rather than copying them, so this can be painful (depending on how difficult the merges were originally). The rerere subsystem, and the contributed retrain script, can help here.

How to maintain parallel histories

(Start, as always, with the git fetch to pick up their commits.)

Use git cherry-pick to copy their commits. That's really all there is to it. You generally don't want rebase as it does a branch name change at the end, and you want to avoid that part. You can do the rebase from a detached HEAD starting point, or make some sacrificial branch names that you can delete after the rebase finishes. Otherwise this is much the same as the undo option: we've just changed the set of commits to copy; instead of copying X, you're copying D.

Merging the histories

(Start, as always, with the git fetch to pick up their commits.)

This is mostly a matter of one-time pain in terms of coming up with the merge result. Having run git fetch, you now have the history I drew:

A  -- B --  C  -- D   <-- originalrepo/main

A# -- B# -- C# -- X   <-- main

While on main, you run git merge --allow-unrelated-histories originalrepo/main to kick off the merging process. This probably fails with lots of conflicts. You then spend hours or days resolving the conflicts, producing the correct merge result, and commit the result:

A---B--C---D   <-- originalrepo/main
            \
A#-B#-C#--X--M   <-- main

New commits you make, and new commits they make, now have a common ancestor, namely commit D:

A---B--C---D--E   <-- originalrepo/main
            \
A#-B#-C#--X--M--F   <-- main

The history goes F-M-(D and X) from your commit F, and E-D from their commit E, so commit D is on both branches and is the merge base. For this future merge, Git will compare the snapshot in D to those in both E and F, and combine the work to make a new merge commit.

Note that your repository and any clones now carries around both histories; theirs carries around only the original history.

Related