nodeJS: convert response.body in utf-8 (from windows-1251 encoding)

user_2935987 0
Viewed 563

I'm trying to convert an HTML body encoded in windows-1251 into utf-8 but I still get messed up characters on html.

They are basically Russian alphabet but I can't get them to be shown properly. I get ??????? ?? ???

const GOT = require('got') // https://www.npmjs.com/package/got
const WIN1251 = require('windows-1251') // https://www.npmjs.com/package/windows-1251

async function query() {
    var body = Buffer.from(await GOT('https://example.net/', {resolveBodyOnly: true}), 'binary')
    var html = WIN1251.decode(body.toString('utf8'))
    console.log(html)
}

query()

enter image description here

node.js encoding
1 Answers

You’re doing a lot of silly encoding back-and-forth here. And the ‘backs’ don’t even match the ‘forths’.

First, you use the got library to download a webpage; by default, got will dutifully decode response texts as UTF-8. You stuff the returned Unicode string into a Buffer with the binary encoding, which throws away the higher octet of each UTF-16 code unit of the Unicode string. Then you use .toString('utf-8') which interprets this mutilated string as UTF-8 (in actuality, it is most likely not valid UTF-8 at all). Then you pass the ‘UTF-8’ string to the windows-1251, to decode it as a ‘code page 1251’ string. Nothing good can possibly come from all this confusion.

The windows-1251 package you want to use takes so-called ‘binary’ (pseudo-Latin-1) strings as input. What you should do instead is take the binary response, interpret it as Latin-1/‘binary’ string and then pass it to the windows-1251 library for decoding.

In other words, use this:

const GOT = require('got');
const WIN1251 = require('windows-1251');

async function query() {
    const body = await GOT('https://example.net/', {
        resolveBodyOnly: true,
        responseType: 'buffer'
    });
    const html = WIN1251.decode(body.toString('binary'))
    console.log(html)
}

query()
user_38401700
Related