How to truncate UTF8 string in JavaScript without breaking multibyte characters?

user_19634938 0
Viewed 43

My apologies if this has been answered here already. I couldn't find a relevant thread.

I'm brand new at JavaScript. I need to be able to truncate a string in a UTF8-friendly way. Currently, using slice() and similar methods, multibyte characters at the end of the string are chopped in half and I get invalid characters.

//Slicing Emojis
var emojitext = "";
var chopped_emoji = emojitext.slice(0, 1);
document.getElementById("slice").innerHTML = chopped_emoji + " is broken";
<p id="slice"></p>

The above code results in an invalid character stored in chopped_emoji. How do I make sure this does not happen?

truncation utf-8 unicode javascript string
1 Answers

JS has a new segmentation API for working with string characters. Depending on the browser support you need, it could be a good fit (you could also polyfill it). You can use it to create an array of graphemes (ie: the visually perceived letters/units on the screen). You can then use .slice() on that array and then .join() it back into a string:

const str = "";

const segmenter = new Intl.Segmenter("en", {granularity: 'grapheme'});
const segItr = segmenter.segment(str);
const segArr = Array.from(segItr, ({segment}) => segment);
const choppedEmoji = segArr.slice(0, 1).join('');

// Examples
console.log("With segmentation", choppedEmoji);
console.log("Without segmentation", str.slice(0, 1));

The above deals with a lot of edge cases such as surrogate pairs (emojis such as ), characters with zero-width joiners (for emojis such as ‍‍), characters with variation selectors (such as ❤️) and composite pair characters (such as ñ).

For simpler characters, you can use Array.from() or the spread syntax (...) to convert your string into an array grouped by the code points of your string (an emoji can consist of multiple code units that encode a single code point), this way when you split, you can avoid splitting on one of the code units (which causes the error character):

const emojitext = "";
const choppedEmoji = [...emojitext].slice(0, 1).join('');
console.log(choppedEmoji + " is not broken :)");

But if you throw something like ‍‍ into the above example though, you'll find that it doesn't work as expected, as ‍‍ is a grapheme that consists of multiple code points joined by zero-width joiner characters, so the above approach won't work with it, but the segmenter option will (same thing with characters consisting of variation selectors and composite characters).

user_56489540
Related