git not detecting single-bit error in a tracked file

Git has the correct data stored. But when you have previously fetched a tree it does not have a way to verify the files are correct.

There are a number of ways this can happen, but most boil down to on-disk bit-rot. If your filesystem has checksum system (zfs) you can be notified of this when you read the file (if you have no redundancy) or when you perform monthly scrubs (with redundancy).

You shouldn't be able to download the error (even with a tcp/networking double bit error) because git should verify the checksum when downloaded.

Use the --full option

This question is related: somebody is deliberately modifying file contents in the object database (as a test) and expecting git to find it. It was not finding the corruption, until they added the --full arg. See this comment below the accepted answer for explicit verification.

What puzzles me is that the official docs state that "full" is now the default, which is why I didn't suggest this earlier. I did not look to see when that changed, but I suppose it's possible that you're using a version of git that does not default to "full" mode.

I suppose it is also possible that git treats implied "full" differently than explicit "full," although I consider that very unlikely.

Thanks to insightful comments from @rodrigo, @Tom and @j6t, I believe this question has now been answered.

How did the bit flip?

There are many potential causes of single-bit errors (a.k.a. "bit rot") on hard drives. This question, and the related links, give a good summary. Importantly, the drive does not need to be failing for single-bit errors to occur. As in this case, the S.M.A.R.T. system and tools such as fsck or chkdsk can report no problems, both before and after such errors occur.

It is also possible that the bit did not flip on the hard drive itself, but rather in a cache somewhere in volatile memory, perhaps as a result of a single-event upset. This could explain why the bit appeared to flip back after some time, when the cache was refreshed.

Why didn't git notice?

The error most likely happened in the working tree, and git did not notice because the file's stat information (size, modification time, etc) had not changed.

Since hashing the contents of every file in a large repository can be time-consuming, git "cheats" by serving cached status info based on metadata, rather than always looking at file contents. This means that a change of data content without a corresponding metadata update will fly under git's radar and go unchecked.

This can be easily verified as follows:

$ echo "All good" > file.txt

$ git add file.txt && git commit -am "Good file"
[main 1a2b3c4d] Good file
 1 file changed, 1 insertion(+), 1 deletion(-)

$ touch -r file.txt /tmp/timestamp-backup

$ echo 'CORRUPT!' > file.txt

$ touch -r /tmp/timestamp-backup file.txt

$ git log --oneline | head -1
1a2b3c4d Good file

$ git status
On branch main
nothing to commit, working tree clean

$ cat file.txt 
CORRUPT!

Once in this state, git will not notice that the file is corrupt until it is modified again, either in the working tree or by checking out a commit that changes it.

Note that if the error had occurred in git's object database (.git/objects), then running git fsck likely would have detected it (see this question for more details).

How can such errors be detected reliably?

On systems where data integrity is important, the possibility of such an error going unnoticed may be unacceptable.

The best solution is to use hardware (e.g. RAID arrays) and/or filesystems (e.g. zfs) that are designed to prevent silent data corruption.

If changing hardware/filesystems is not an option, then for git repositories specifically:

1. Run git fsck to verify the object database.

2. Run git ls-files -z | xargs -0 touch to force git to re-index the working tree. This command changes the timestamp of every tracked file so that next time git status is run, git will re-hash every file and pick up any changes.