Extraneous character while converting string to buffer and back in javascript

user_1583052 0
Viewed 46

In node v16.14.2,

> String.fromCharCode.apply(null, Buffer.from('°'))
'°'

This works fine though:

> Buffer.from('°').toString('utf-8')
'°'

I want to know why the first scenario adds an extra character in the output.

Browser example:

console.log(String.fromCharCode.apply(null, ethereumjs.Buffer.Buffer.from('°'))) /// '°'
console.log(ethereumjs.Buffer.Buffer.from('°').toString('utf-8')) // '°'
console.log(new Uint8Array(ethereumjs.Buffer.Buffer.from('°'))) // [ 194, 176 ]
console.log(new Uint8Array(ethereumjs.Buffer.Buffer.from(String.fromCharCode(176)))) // [ 194, 176 ]
<script src="https://cdn.jsdelivr.net/gh/ethereumjs/browser-builds/dist/ethereumjs-tx/ethereumjs-tx-1.3.3.min.js"></script>

EDIT: Another funny thing:

> new Uint8Array(Buffer.from('°'))
Uint8Array(2) [ 194, 176 ]
> new Uint8Array(Buffer.from(String.fromCharCode(176)))
Uint8Array(2) [ 194, 176 ]
node.js javascript
1 Answers

The default string encoding of Buffer.from is UTF-8 which encodes characters in the Extended ASCII range using two bytes. (It can't just use one byte because the highest order bit of that byte is used to signal multibyte encoding.)

UTF-8 two-byte encoding has the form:

110xxxxx 10xxxxxx

which gives 11 bits to represent a character code.

The ASCII code of '°' is 176

> '°'.charCodeAt(0)
176

which in binary is:

> (176).toString(2)
'10110000'

So replacing the x with this binary value we have:

110xxxxx 10xxxxxx
110xxx10 10110000
      ^^   ^^^^^^

which means the UTF-8 two-byte encoding of '°' is

11000010 10110000

or

194 176

as expected:

> [...Buffer.from('°')]
[ 194, 176 ]

> String.fromCharCode(194)
'Â'

> String.fromCharCode(176)
'°'
user_15655120
Related