Git: how to import commits from another repository and unsquash them?

It is impossible to do without rewriting the history of the new repository, thus breaking users of the new repo that e.g. have forked the repo and are working on their own branches. The hash of the head commit includes the entire history of the repository, so if you tack on the "old" history to the front, you will need to rewrite all commits afterwards, thus changing their hash.

However, you could graft the old history to the new repository, or even better you could use git replace.

On the other hand, if you are willing to abandon the new repository, then it is simply a matter of replaying the commits on the old repository. You could do this in the maximally stupid way of literally just checking out the new repo on top of the old repo commit-by-commit, grab the message, author name and date, committer name and date, and commit the result, then repeat.

Importing commits is easy: just use git fetch. Each commit has its own unique ID and your Git and the other Git you're talking to (via git fetch) will tell each other their IDs, so that your Git can get any commits they have that you don't.

The hard part is stringing together commits that have no relationships: if their commits don't refer, by number, to yours, and vice versa, then your commits and theirs are unrelated.

Details, starting with fetch

I'm now in a situation where I need to find a way to "import" 1100+ commits from the source repository, preferably without losing commit history after it ...

Commit history is simply the commits. Each commit has its own unique ID (I know I said that already, but it's important), and each commit refers back to some earlier commit(s) by ID-number.

If we draw this using uppercase letters to stand in for actual commit hash ID numbers, we get pictures that look like this:

A <-B <-C ... <-G <-H   <--main

Here, your Git uses your branch name main to find the actual hash ID of the last commit in the chain, which we're calling H. Commit H refers back to earlier commit G, which refers back to another still-earlier commit, and so on. With just eight commits in your repository, this backwards-looking chain, as read by Git starting at the end and working backwards, ends (starts?) at commit A.

If these eight commits were created from the same source snapshot as some other series of commits, but without actually using the hash ID from the other Git, then some other Git repository has another, unrelated chain:

K <-L <-M ... <-P   <--somebranch

So their Git repository uses the name somebranch to find the last commit in their chain; that's commit P, and it points back to earlier commits, which continues until their chain starts/ends at their first-ever commit, which we're calling K (having skipped I and J for no apparent reason).

It sounds like you're saying that the snapshot in P matches your snapshot in A. It's easy to tell whether that's the case because you can find your first-ever commit by having your Git walk from your last commit (H) backwards to your first. Then, with that commit hash ID in your cut-and-paste buffer, you run:

git remote add xyzzy <url-for-their-git>

(pick some more meaningful name than xyzzy here), and then you run:

git fetch xyzzy
git diff <hash-of-A> xyzzy/somebranch

Replace xyzzy and somebranch with the appropriate name that finds commit P. Remember that your Git got their commits during the git fetch, including commit P at the end of their chain; then your Git copied their name somebranch to your xyzzy/somebranch remote-tracking name, to let you easily find commit P in your repository. So the git diff above compares the snapshot in your first commit to the one in their last commit.

(Even if the snapshots are different, that's not fatal. You might want to treat things a little differently in that case, though: perhaps, find some pre-P commit in their repository that does match, or, later compensate for whatever different here, if that's appropriate and reasonable.)

But the thing is, after this git fetch, these two chains are not related. You have:

A--B--...--H   <-- main

K--L--...--P   <-- xyzzy/somebranch

in your repository. Because Git commit hash IDs are truly unique to each commit, and no part of any commit can ever be changed, you literally cannot hook these chains together.

Hooking them together

I just said you can't hook them together, and that's literally true, but there are two things you can do. One involves coming up with a sort of temporary replacement, and you'll probably want to do that even if you make it permanent. The other involves coming up with permanent replacements.

Git supports, as a general concept, the idea of making a sort of lookaside replacement commit. (In fact you can do this for any internal Git object at all, but commits are the ones that you want, here.) You use the git replace command to build these. They create a new object—in this case, a new commit object—and write a sort of side instruction to Git that says: If you're about to use commit X, use this other commit X' instead. So we pick one of your existing commits—perhaps B, if A and P match up exactly so that we don't really need to keep A at all—and tell Git: when you're about to use commit B, use commit B' instead.

What we have to do is construct a commit B' that looks a whole heck of a lot like B, with one change. Instead of B''s parent being A, we choose P. The result looks like this:

          A--B--C--...--H   <-- main
             :
             B'   <-- refs/replace/<hash-of-B>   # special name
            /
K--L--...--P   <-- xyzzy/somebranch

Now, when we ask your Git to start at H and work backwards, here's what happens:

• show/use commit H
• show/use commit G
• ...
• show/use commit C
• show/use commit WHOOP! WHOOP! REPLACEMENT ALERT! SWITCH TO B'
• show/use commit P
• show/use commit O
• ...

and now it seems as though commits start at K and run up through and end at H.

They don't—there's just this funky replacement thing—but most of Git obeys the replacement. The git log command can detect the swap-out and include a marker, and any Git command can be told to ignore the replacements by running git --no-replace-objects command. But mostly, most Git commands obey the replacement.

Note that if A and P don't match, source-code-snapshot-wise, you'll want to use git replace to replace A with A', rather than replacing B with B'. In either case, you want git replace --graft; see the git replace documentation for details.

Replacements don't clone

The drawback to the replacement trick is that git clone and similar operations tend to completely ignore the replacement. That's why I said most of Git obeys it: clone deliberately doesn't. There are some ways to clone the replacement as well, manually after an initial clone, but they're clunky to use and you probably don't want to do that.

Making the replacement permanent

Once you have the replacement in place, and git log and other operations all look good, you can have Git "rewrite history", using the old git filter-branch or the newfangled git filter-repo. Just have them do the rewrite without changing anything, but also without disabling replacement. These operations (filter-branch or filter-repo) copy commits to new-and-presumably-improved commits. If and when the new copy is bit-for-bit identical to the original, it gets the same number as the original, so the commits we have been calling K through P here, and finding via xyzzy/somebranch, won't change.¹

Because it has to, this copy operation goes forwards, unlike most of Git. Most of Git works backwards, from the end to the beginning. Even filter-branch / filter-repo have to list the commits backwards; they just then make sure to do the copying backwards from the listing, so that it's forwards.

Now, note that when copying commit B (or A), your Git will go in and copy instead B' (or A'). It will retain all its data because nothing has changed here either. But then after that, Git will copy C (or B)—but this time, the parent of the new copy obeys the replacement rules. So the parent is the B' (or A') replacement: C (or B) becomes C' (or B'). This then repeats for each subsequent commit. The end result is this, assuming that B' is the replacement for B:

          A--B--C--...--H
             
             B'-C'-...'-H'
            /
K--L--...--P

The last step of one of these filter operations—branch or repo—is to take the branch names² and make them point at the corresponding copy. So xyzzy/somebranch still points to P, because the new copy of P is P, but the name main now points to H', because that's the new copy of H:

          A--B--C--...--H   ???
             
             B'-C'-...'-H'  <-- main
            /
K--L--...--P   <-- xyzzy/somebranch

Commit H is no longer findable.³ As such, you just won't see it any more, for a while, and eventually, Git will notice that it's not only not being used, it can't be used. Git will then (eventually) remove it entirely and you will be left with a history that joins up the way you want.

¹At least, that's the goal. There was a bug in filter-branch that turned up on MacOS once. It has been fixed. There might be other bugs that might turn up. If these hash IDs do change, you've found a new bug.

²With git filter-branch, if you have tag names, it is important to add --tag-name-filter cat to make it update tag names too. I have no actual experience with git filter-repo yet but it should probably be more sensible about this. The filter-branch command probably won't update any remote-tracking names, but in this example, it makes no difference whether it does or not.

³With filter-branch, there are some refs/original/ names that find the original commits. You have to clean those up manually afterward, once you're satisified that everything looks right. Consult the documentation for details.