How to find out why a file has been deemed text or binary by git for EOL conversion?

user_1797006 0
Viewed 319

If a .gitattributes is present in the repository or a configuration option is set to enable end-of-line (EOL) conversion, git needs to make a decision whether a file is text or binary.

Sometimes this decision is not obvious, e.g. if invisible characters are present in the file, see https://confluence.atlassian.com/bbkb/file-detected-as-binary-not-displayed-as-text-in-bitbucket-892611499.html for an example.

The presence of characters which cause the file to recognized as something that it is not, is something that you might want to fix in most cases. However, the analysis with hexdump and vi as proposed in the linked post can be exhaustive and for some files and/or users practially impossible. Is there a way to find out what causes git to recognize a file as text or binary in a verbose matter (e.g. "reconized [path] as binary because of presence of [some codepoint] at line [n]")?

Our team is using Git 2.19 and 2.17 on Ubuntu 18.10, Windows 10 and macOS.

line-endings eol git
2 Answers

git relies on buffer_is_binary in its xdiff-interface.c file to decide whether a file is binary or text. That function is called from Git's merging code, among other places. The logic is straightforward - a file is binary if there's a 0 byte in the first 8000 bytes of it. The code is:

#define FIRST_FEW_BYTES 8000
int buffer_is_binary(const char *ptr, unsigned long size)
{
    if (FIRST_FEW_BYTES < size)
        size = FIRST_FEW_BYTES;
    return !!memchr(ptr, 0, size);
}

As such, you can have very simple files being detected as binary if they're encoded in UTF-16, which is a common reason for Git treating files as binary. A text file containing

a b

will be detected as binary if saved in UTF-16 because its hexdump output is, with the LF file ending:

0000000 6100 2000 6200 0a00

For instance, the space (0x20 in ASCII or UTF-8) is encoded as 0x0020 in UTF-16, hence Git considers the file binary.

So a "verbose" mode wouldn't really help much as you need to locate 0 bytes. grep could do that in Perl-regex mode such as grep -obaUP "\x00" filename to print byte offsets of 0-value bytes.

user_12647270

git ls-files --eol displays information about how files are identified by git and how they're committed:

--eol

Show and of files. is the file content identification used by Git when the "text" attribute is "auto" (or not set and core.autocrlf is not false). is either "-text", "none", "lf", "crlf", "mixed" or "".

"" means the file is not a regular file, it is not in the index or not accessible in the working tree.

is the attribute that is used when checking out or committing, it is either "", "-text", "text", "text=auto", "text eol=lf", "text eol=crlf". Since Git 2.10 "text=auto eol=lf" and "text=auto eol=crlf" are supported.

Both the in the index ("i/") and in the working tree ("w/") are shown for regular files, followed by the ("attr/").

from the git ls-files documentation

user_17970060
Related