Find nearest safe split in byte array containing UTF-8 data

user_2382729 0
Viewed 563

I want to split a large array of UTF-8 encoded data, so that decoding it into chars can be parallelized.

It seems that there's no way to find out how many bytes Encoding.GetCharCount reads. I also can't use GetByteCount(GetChars(...)) since it decodes the entire array anyways, which is what I'm trying to avoid.

utf-8 encoding string c#
2 Answers

Great answer Mathieu and Der, adding a python variant 100% based on your answer which works great:

def find_utf8_split(data, bytes=None):

  bytes = bytes or len(data)

  while bytes > 0 and data[bytes - 1] & 0xC0 == 0x80:
    bytes -= 1

  if bytes > 0:
    if data[bytes - 1] & 0xE0 == 0xC0: bytes = bytes - 1
    if data[bytes - 1] & 0xF0 == 0xE0: bytes = bytes - 1
    if data[bytes - 1] & 0xF8 == 0xF0: bytes = bytes - 1

  return bytes

This code finds a UTF-8 compatible split in a given byte string. It does not do the split as that would take more memory, that is left to the rest of the code.

For example you could:

position = find_utf8_split(data)
leftovers = data[position:]
text = data[:position].decode('utf-8')
user_15076490
Related