How to recover Git objects damaged by hard disk failure?

Viewed 107132

I have had a hard disk failure which resulted in some files of a Git repository getting damaged. When running git fsck --full I get the following output:

error: .git/objects/pack/pack-6863e0a0e4b4ded6090fac5d12eba6ca7346b19c.pack SHA1 checksum mismatch
error: index CRC mismatch for object 6c8cae4994b5ec7891ccb1527d30634997a978ee from .git/objects/pack/pack-6863e0a0e4b4ded6090fac5d12eba6ca7346b19c.pack at offset 97824129
error: inflate: data stream error (invalid code lengths set)
error: cannot unpack 6c8cae4994b5ec7891ccb1527d30634997a978ee from .git/objects/pack/pack-6863e0a0e4b4ded6090fac5d12eba6ca7346b19c.pack at offset 97824129
error: inflate: data stream error (invalid stored block lengths)
error: failed to read object 0dcf6723cc69cc7f91d4a7432d0f1a1f05e77eaa at offset 276988017 from .git/objects/pack/pack-6863e0a0e4b4ded6090fac5d12eba6ca7346b19c.pack
fatal: object 0dcf6723cc69cc7f91d4a7432d0f1a1f05e77eaa is corrupted

I have backups of the repository, but the only backup that includes the pack file has it already damaged. So I think that I have to find out a way to retrieve the single objects from different backups and somehow instruct Git to produce a new pack with only correct objects.

Can you please give me hints how to fix my repository?

8 Answers

In some previous backups, your bad objects may have been packed in different files or may be loose objects yet. So your objects may be recovered.

It seems there are a few bad objects in your database. So you could do it the manual way.

Because of git hash-object, git mktree and git commit-tree do not write the objects because they are found in the pack, then start doing this:

mv .git/objects/pack/* <somewhere>
for i in <somewhere>/*.pack; do
  git unpack-objects -r < $i
done
rm <somewhere>/*

(Your packs are moved out from the repository, and unpacked again in it; only the good objects are now in the database)

You can do:

git cat-file -t 6c8cae4994b5ec7891ccb1527d30634997a978ee

and check the type of the object.

If the type is blob: retrieve the contents of the file from previous backups (with git show or git cat-file or git unpack-file; then you may git hash-object -w to rewrite the object in your current repository.

If the type is tree: you could use git ls-tree to recover the tree from previous backups; then git mktree to write it again in your current repository.

If the type is commit: the same with git show, git cat-file and git commit-tree.

Of course, I would backup your original working copy before starting this process.

Also, take a look at How to Recover Corrupted Blob Object.

Banengusk was putting me on the right track. For further reference, I want to post the steps I took to fix my repository corruption. I was lucky enough to find all needed objects either in older packs or in repository backups.

# Unpack last non-corrupted pack
$ mv .git/objects/pack .git/objects/pack.old
$ git unpack-objects -r < .git/objects/pack.old/pack-012066c998b2d171913aeb5bf0719fd4655fa7d0.pack
$ git log
fatal: bad object HEAD

$ cat .git/HEAD 
ref: refs/heads/master

$ ls .git/refs/heads/

$ cat .git/packed-refs 
# pack-refs with: peeled 
aa268a069add6d71e162c4e2455c1b690079c8c1 refs/heads/master

$ git fsck --full 
error: HEAD: invalid sha1 pointer aa268a069add6d71e162c4e2455c1b690079c8c1
error: refs/heads/master does not point to a valid object!
missing blob 75405ef0e6f66e48c1ff836786ff110efa33a919
missing blob 27c4611ffbc3c32712a395910a96052a3de67c9b
dangling tree 30473f109d87f4bcde612a2b9a204c3e322cb0dc

# Copy HEAD object from backup of repository
$ cp repobackup/.git/objects/aa/268a069add6d71e162c4e2455c1b690079c8c1 .git/objects/aa
# Now copy all missing objects from backup of repository and run "git fsck --full" afterwards
# Repeat until git fsck --full only reports dangling objects

# Now garbage collect repo
$ git gc
warning: reflog of 'HEAD' references pruned commits
warning: reflog of 'refs/heads/master' references pruned commits
Counting objects: 3992, done.
Delta compression using 2 threads.
fatal: object bf1c4953c0ea4a045bf0975a916b53d247e7ca94 inconsistent object length (6093 vs 415232)
error: failed to run repack

# Check reflogs...
$ git reflog

# ...then clean
$ git reflog expire --expire=0 --all

# Now garbage collect again
$ git gc       
Counting objects: 3992, done.
Delta compression using 2 threads.
Compressing objects: 100% (3970/3970), done.
Writing objects: 100% (3992/3992), done.
Total 3992 (delta 2060), reused 0 (delta 0)
Removing duplicate objects: 100% (256/256), done.
# Done!

Git checkout can actually pick out individual files from a revision. Just give it the commit hash and the file name. More detailed info here.

I guess the easiest way to fix this safely is to revert to the newest uncommited backup and then selectively pick out uncorrupted files from newer commits. Good luck!

The solution by Daniel Fanjul looked promissing. I was able to find that blob file and extracted it ("git fsck --full --no-dangling", "git cat-file -t {hash}", "git show {hash} > file.tmp") but when I tried to update pack file with "git hash-object -w file.tmp", it displayed correct hash BUT the error remained.

So I decided to try different approach. I could simply delete local repository and download everything from remote but some branches in local repository were 8 commits ahead and I did not want to lose those changes. Since that tiny, 6kb mp3 file, I decided to delete it completely. I tried many ways but the best was from here: https://itextpdf.com/en/blog/technical-notes/how-completely-remove-file-git-repository

I got the file name by running this command "git rev-list --objects --all | grep {hash}". Then I did a backup (strongly recommend to do so because I failed 3 times) and then run the command:

"java -jar bfg.jar --delete-files {filename} --no-blob-protection ."

You can get bfg.jar file from here https://rtyley.github.io/bfg-repo-cleaner/ so according to documentation I should run this command next:

"git reflog expire --expire=now --all && git gc --prune=now --aggressive"

When I did so, I got errors on last step. So I recovered everything from backup and this time, after removing file, I checkout to the branch (which was causing that error), then check out back to main and only after run the command one after each other:

"git reflog expire --expire=now --all" "git gc --prune=now --aggressive"

Then I added my file back to its location and comit. However, since many local commits were changed, I was not able to push anything to server. So I backup everything on server (in case I screw it), check out to the branch which was affected and run the command "git push --force".

What I understood from this case? GIT is great but so senstive... I should have an option to simply disregard one f... 6kb file I know what I am doing. I have no clude why "git hash-object -w" did not work either =( Lessons learnt, push all commits, do not wait, do backup of repository time to time. Also I know how to remove files from repository, if I ever need =)

I hope this saves someone's time

Related