Compatibility of printf with utf-8 encoded strings

user_8795485 0
Viewed 1521

I'm trying to format some utf-8 encoded strings in C code (char *) using the printf function. I need to specify a length in format. Everything goes well when there are no multi-bytes characters in parameter string, but the result seems to be incorrect when there are some multibyte chars in data.

my glibc is kind of old (2.17), so I tried with some online compilers and result is the same.

#include <stdlib.h>
#include <locale.h>

int main(void)
{
    setlocale( LC_CTYPE, "en_US.UTF-8" );
    setlocale( LC_COLLATE, "en_US.UTF-8" );

    printf( "'%-4.4s'\n",   "elephant" );
    printf( "'%-4.4s'\n",   "éléphant" );
    printf( "'%-20.20s'\n", "éléphant" );

    return 0;
}

Result of execution is :

'elep'
'él�'
'éléphant          '

First line is correct (4 chars in output)

Second line is obviously wrong (at least from a human point of view)

Last line is also wrong : only 18 unicode chars are written instead of 20

It seems that the printf function count chars before UTF-8 decoding (counting bytes instead of unicode chars)

Is that a bug in glibc or a well documented limitation of printf ?

glibc utf-8 printf unicode gcc
2 Answers

It's true that printf counts bytes, not multibyte characters. If it's a bug, the bug is in the C standard, not in glibc (the standard library implementation usually used in conjunction with gcc).

In fairness, counting characters wouldn't help you align unicode output either, because unicode characters are not all the same display width even with fixed-width fonts. (Many codepoints are width 0, for example.)

I'm not going to attempt to argue that this behaviour is "well-documented". Standard C's locale facilities have never been particularly adequate to the task, imho, and they have never been particularly well documented, in part because the underlying model attempts to encompass so many possible encodings without ever grounding itself in a concrete example that it is almost impossible to explain. (...Long rant deleted...)

You can use the wchar.h formatted output functions, which count in wide characters. (Which still isn't going to give you correct output alignment but it will count precision the way you expect.)

user_15662210

Let me quote rici: It's true that printf counts bytes, not multibyte characters. If it's a bug, the bug is in the C standard, not in glibc (the standard library implementation usually used in conjunction with gcc).

However, don't conflate wchar_t and UTF-8. See wikipedia to grasp the sense of the former. UTF-8, instead, can be dealt with almost as if it were good old ASCII. Just avoid truncating in the middle of a character.

In order to get alignment, you want to count characters. Then, pass the bytes count to printf. That can be achieved by using the * precision and passing the count of bytes. For example, since accented e takes two bytes:

    printf("'-4.*s'\n", 6, "éléphant");

A function to count bytes is easily coded based on the format of UTF-8 characters:

    static int count_bytes(char const *utf8_string, int length)
    {
        char const *s = utf8_string;
        for (;;)
        {
            int ch = *(unsigned char *)s++;
            if ((ch & 0xc0) == 0xc0) // first byte of a multi-byte UTF-8
                while (((ch = *(unsigned char*)s) & 0xc0) == 0x80)
                    ++s;
            if (ch == 0)
                break;
            if (--length <= 0)
                break;
        }
        return s - utf8_string;
    }

At this point however, one would end up with lines like so:

    printf("'-4.*s'\n", count_bytes("éléphant", 4), "éléphant");

Having to repeat the string twice quickly becomes a maintenance nightmare. At a minimum, one can define a macro to make sure the string is the same. Assuming the above function is saved in some utf8-util.h file, your program could be rewritten as follows:

    #include <stdio.h>
    #include <stdlib.h>
    #include <locale.h>
    #include "utf8-util.h"

    #define INT_STR_PAIR(i, s) count_bytes(s, i), s
    int main(void)
    {
        setlocale( LC_CTYPE, "en_US.UTF-8" );
        setlocale( LC_COLLATE, "en_US.UTF-8" );

        printf( "'%-4.*s'\n",  INT_STR_PAIR(4, "elephant"));
        printf( "'%-4.*s'\n",  INT_STR_PAIR(4, "éléphant"));
        printf( "'%-4.*s'\n",  INT_STR_PAIR(4, "ééphant"));
        printf( "'%-20.*s'\n", INT_STR_PAIR(20, "éléphant"));

        return 0;
    }

The last but one test uses , the Greek acrophonic thespian three hundred (U+1016B) character. Given how the counting works, testing with consecutive non-ASCII characters makes sense. The ancient Greek character looks "wide" enough to see how much space it takes using a fixed-width font. The output may look like:

    'elep'
    'élép'
    'éép'
    'éléphant          '

(On my terminal, those 4-char strings are of equal length.)

user_37456920
Related