Lets say that I have a array of UTF-16 bytes. How would I count how many characters are in the array of UTF-16 bytes? The array can also be inside of a boundary. For example, lets say that there is a 4 byte UTF-16 character, and only 3 of the 4 bytes are read into a buffer. I then try counting that 3 byte buffer. How would I detect that there is not enough bytes?
2 Answers
4-byte UTF-16 characters are called surrogate pairs, and all surrogate pairs start with a code unit in the range 0xD800 to 0xDBFF inclusive. To count the number of characters (aka code points) in a UTF-16 string you therefore want to do something like this (pseudo-code):
char_count = 0;
string_pos = 0;
while (!end_of_string)
{
code_unit = input_string [string_pos];
++char_count;
if (code_unit >= 0xd800 && code_unit <= 0xdbff)
string_pos += 2;
else
++string_pos;
}
To detect an incomplete surrogate pair, just check if there are any code units left in the string after detecting the lead-in value. You might also want to check for invalid surrogate pairs.
Wikipedia has a good write-up on UTF-16 here.
Examine the state of the decoder. If the state is READY then it is enough, otherwise it is not enough. Of course, you need to maintain the state with each incoming code point.
Related