web-scraping fix bad character symbol

Question

to begin, i'm french so sorry if my english isn't perfect.

I'm role player and i need to scraper this http://www.gemmaline.com/sorts/liste-classe-pretre.htm which is in Iso-8859-1 to catch each text list and later to catch each information from each link from name list. i have already formulas on google sheet to scraper it. But like lot of know, importxml in google sheet it's 50 request max, and it's very slow. So i try a different process with javascript and node js. To use axios and cheerio to scraper. It works but the result is uncorrect for each accented character or single quote. And after lot of try i didn't solve my issue. This is the code:

const PORT = 8000
const axios = require('axios')
const cheerio = require('cheerio')
const express = require('express')
const fs = require('fs')

const app = express()

const url = 'http://www.gemmaline.com/sorts/liste-classe-pretre.htm'

axios(url)
    .then(response => {
        const html = response.data
        const $ = cheerio.load(html)
        const data = []

        $('body:nth-child(2) ul li').each(function() {
            
            //const encoder = new TextEncoder()
            //const name = new TextDecoder().decode(new Uint8Array(encoder.encode($(this).text())))

            const name = $(this).text()
            const url = $(this).find('a').attr('href')

            data.push({ name, url})
        })

        fs.writeFileSync('test.json', JSON.stringify(data, null, 1))
    }).catch(err => console.log(err))

    

app.listen(PORT, ()=>console.log(`server running on port ${PORT}`))

and this is my result in a file, btw it's the same if i did just a console.log(data):

result

Now you can see strange symbol. If someone know how to fix it, i will be really happy.

LaCerise · Answer

it works !

const PORT = 8000
const axios = require('axios')
const cheerio = require('cheerio')
const express = require('express')
const fs = require('fs')

const app = express()

const url = 'http://www.gemmaline.com/sorts/liste-classe-pretre.htm'


axios(url, {
    
    responseEncoding: 'binary'
})
    .then(response => {

        const $ = cheerio.load(response.data.toString('ISO-8859-1'),{decodeEntities: false})
        const data = []

        $('body:nth-child(2) ul li').each(function() {

            const name = $(this).text()
            const url = $(this).find('a').attr('href')

            data.push({ name, url })
        })

        fs.writeFileSync('test2.json', JSON.stringify(data, null, 1))
    }).catch(err => console.log(err))


app.listen(PORT, () => console.log(`server running on port ${PORT}`))

wrong character line 11

I have now a new issue, a single quote at line 11 should be not become ’. On this line it's normaly Conservation d'organe Nécromancie instead of Conservation dorgane Nécromancie. After, i could just substitute this symbol by a single quote for each time i get it but maybe we can find something more clean directly by encoding

web-scraping fix bad character symbol

1 Answers