Unicode character order problem when text is displayed

user_16835081 0
Viewed 63

I am working on an application that converts text into some other characters of the extended ASCII character set that gets displayed in custom font.

The program operation essentially parses the input string using regex, locates the standard characters and outputs them as converted before returning a string with the modified text which displays correctly when viewed with the correct font.

Every now and again, the function returns a string where the characters are displayed in the wrong order, almost like they are corrupted or some data is missing from the Unicode double width spacing. I have examined the binary output, the hex data, and inspected the data in the function before i return it and everything looks ok, but every once in a while something goes wrong and cant quite put my finger on it.

To see an example of what i mean when i say the order is weird, just take a look at the following piece of converted text output from the program and try to highlight it with your mouse. You will see that it doesn't highlight in the order you expect despite how it appears.

Has anyone seen anything like this before and have they any ideas as to what is going on?

ך┼♫יἯ╡П♪דἰ

unicode text
1 Answers

You are mixing various Unicode characters with different LTR/RTL characteristics.

LTR means "left-to-right" and is the direction that English (and many other western language) text is written.

RTL is "right-to-left" and is used mostly by Arabic and Hebrew (as well as several other scripts).

By default when rendering Unicode text the engine will try to use the directionality of the characters to figure out what direction a given part of the code should go. And normally that works just fine because Hebrew words will have only Hebrew letters and English words will only use letters from the Latin alphabet, so for each chunk there's a easily guessable direction that makes sense.

But you are mixing letters from different scripts and with different directionality.

For example ך is U+05DA HEBREW LETTER FINAL KAF, but you also use two other Hebrew characters. You can use something like this page to list the Unicode characters you used.

You can either

  1. not use "wrong" directionality letters or
  2. make the direction explict using a Left-to-right mark character.

Edit: Last but not least: I just realized that you said "custom font": if you expect displaying with a specific custom font, then you should really be using one of the private use areas in Unicode: they are explicitly reserved for private use like this (i.e. where the characters don't match the publicly defined glyphs for the codepoints). That would also avoid surprises like the ones you get, where some of the used characters have different rendering properties.

user_403420
Related