What are the risks of splitting a unicode string?

Viewed 42

I understand that a UTF-8 encoded string should not be split in the middle of a multi-byte representation of a unicode scalar (a code point). Similarly for UTF-16.

At a higher level, a unicode string is a sequence of code points (unicode scalars), and clusters of code points can be used to form complex graphemes, for example the code points स and ् form the grapheme स्. So a string should not be split in the middle of such a cluster. Similarly, flags are represented using a sequence of Regional Indicator Symbols, so one should be careful not to split the string in the middle of such a sequence.

Also, there are left-to-right (LTR) and right-to-left (RTL) code points that are useful for bidirectional text. I suppose that when a unicode string containing bidirectional text is split in two parts, we need to insert extra LTR or RTL code points in the second part to preserve the original directionality, is that right?

And perhaps, if the original string contains a Byte-Order-Mark (BOM) code point, it should also be added to the second part, is that right?

Are there other things one should be careful about when splitting Unicode strings? I'm trying to evaluate how well various programming languages support Unicode.

Edit: Adriano Repetti's comment below points to Annex #29 from the Unicode Standard, which answers this question in detail (thanks!).

1 Answers

The following was a comment from Adriano Repetti:

I already wrote about it in a few places then let me just recap: even if you limit yourself to splitting code units (then I won't consider broken encoding, surrogates & similar, let's leave code points out of the game) there are rules about where a string can be broken. See https://unicode.org/reports/tr29/. Side note: I wouldn't expect to see BOM in any string (it's only about encoding and only if you're dealing with its byte representation in a storage) – Adriano Repetti

Related