How does concatting these two strings lead to this result?

user_10434973 0
Viewed 117

I've written the code below to illustrate this behaviour that happens when concatting these two string.

const getBytes = x => {
    let buf = Buffer.from(x);
    const n = [];
    
    for (const value of buf.values()) {
        n.push(value);
    }

    return [n, parseInt(buf.toString('hex'), 16)];
};


let x = unescape('%uDB40');
let y = unescape('%uDD31');

console.log(typeof(x), typeof(y));
console.log(Buffer.from(x), getBytes(x), );
console.log(Buffer.from(y), getBytes(y));
console.log(Buffer.from(x+y), getBytes(x+y));

The result is:

string string
<Buffer ef bf bd> [ [ 239, 191, 189 ], 15712189 ]
<Buffer ef bf bd> [ [ 239, 191, 189 ], 15712189 ]
<Buffer f3 a0 84 b1> [ [ 243, 160, 132, 177 ], 4087383217 ]

I'm unable to understand how it ends up as a completely different result which is preventing me from successfully porting this behaviour.

emulation hex javascript
1 Answers

As the Buffer.from documentation states: "When converting between Buffers and strings, a character encoding may be specified. If no character encoding is specified, UTF-8 will be used as the default."

To convert from a JavaScript string (UTF-16) to a Buffer (UTF-8), you wrote:

let buf = Buffer.from(x);

If a UTF-16 to UTF-8 character conversion fails, it writes the Unicode replacement character to Buffer, as prescribed by the Unicode Standard.

I used these links to answer your question: Node.js: Buffer, U+DB40, U+DD31, U+FFFD, U+E0131.

For a JavaScript reference: JavaScript: The Definitive Guide, by David Flanagan.

For a Unicode reference: Unicode Standard.

user_164658020
Related