Removed file from git history still exists?

Viewed 61

In one old commit a big file was added. After several commits it was removed however it stay in the history so I am trying to remove this file from history by the following command:

git filter-branch -f --index-filter "git rm -rf --cached --ignore-unmatch src/main/resources/embed/linux/mongodb-linux-x86_64-3.2.0.tgz" --pru-tag-name-filter cat -- --all

the result of this command is following:

Rewrite 9e96a248322ba6e85efde1e41498db0d41c1ee79 (3026/5151) (13917 seconds passed, remaining 9773 predicted)    rm 'src/main/resources/embed/lib-linux-x86_64-3.2.0.tgz'
Rewrite 070cbe84ad8e1bc8f1ccec5294b600218d714e30 (4123/5151) (18963 seconds passed, remaining 4728 predicted)    rm 'src/main/resources/embed/lib-linux-x86_64-3.2.0.tgz'
Rewrite 1f5604b332d6d48169fa398b60fc5ac42124d63d (5151/5151) (24005 seconds passed, remaining 0 predicted)
Ref 'refs/heads/master' was rewritten
Ref 'refs/remotes/origin/master' was rewritten
....

as I understand from the output the file was removed from commit 9e96a248322ba6e85efde1e41498db0d41c1ee79, so I check this:

git checkout 9e96a248322ba6e85efde1e41498db0d41c1ee79
ls src/main/resources/embed/linux/

the result is mongodb-linux-x86_64-3.2.0.tgz. So the file still exist in the history. What I did wrong? How can I remove it compliantly?

2 Answers

as I understand from the output the file was removed from commit 9e96a248322ba6e85efde1e41498db0d41c1ee79

git filter-branch removed the file and created a new commit. The new commit has a different hash because its contents are different. The original commit with the hash 9e96a248322ba6e85efde1e41498db0d41c1ee79 still exists until you run git gc. Note that if you run git log | grep 9e96a, that hash will not appear because the corresponding commit is no longer in your history.

To understand this in more detail, I suggest you read about how a commit is created. This blog article explains it well. And this chapter in Pro Git goes into detail about the internal plumbing of git and how a commit is created.

The answer by code-apprentice is mostly correct, but to elaborate a bit:

It is impossible to modify a commit. What filter-branch (or any history-rewriting mechanism) does is to create new commits that are "just like" existing commits, but with the changes you ask for. (e.g. your new commits don't have the large file in them, but otherwise they look like your old commits)

This has two important consequences.

First, after you rewrite the history of a branch, anyone else with a copy of that branch will need to perform some recovery steps. You can learn about those in the git rebase documentation (under "recovering from upstream rebase"). If they do the wrong thing to recover, it will undo the history rewrite (putting the large file back in the history), so you need everyone to cooperate when rewriting history. For that reason, sometimes it is not practical to rewrite history. For large rewrites (especially those affecting a complex history, multiple branches, etc.) it is often best to arrange a cut-over where everyone discards their existing clones and then re-clones from the rewritten repo.

(A nuance of that, is that any tooling or documentation that depends on commit IDs is also broken when you do a history rewrite.)

Second, it means that doing the rewrite does not immediately remove the old commits from the repo. That's important both if you're trying to recover space from binary bloat, and also if you're trying to clean up a leak of sensitive information into the repo history. (In the latter case, it's almost always necessary to treat the information as compromised.)

Running git gc can be part of the solution, but it isn't sufficient. That's because right after you do the rewrite, your repo still has references to the old commits, so gc doesn't see them as garbage. At a minimum, you'd need to cleanse the reflogs; there may be other issues as well.

There are ways to do that, but generally I sidestep the issue by just making a new clone from the rewritten repo. That clone shouldn't bother copying the old/removed history. (To be clear - I don't think there's any documented requirement that says such a clone can't copy the defunct history, but in my experience - at least when using the standard git implementation - it doesn't.) Then you could destroy the origin repo and recreate it from the fully-sanitized clone.

Related