existing attribute selected by Cheerio sometimes return undefined (get HTML with Puppeteer)

Question

I use Puppeteer to get website HTML and then scrape data with Cheerio. Here is part of my code. It works fine almost every time, but sometimes I get undefined from companyAddress and companyIntro. In the beginning, I thought it may be due to the differences of different pages, but it happens even when I scraped the same page at different times (most of time I get the data but sometimes it is undefined). The page is rendered successfully, and the attributes and their value are confirmed to exist via devtool. I wonder the reason behind it. Could it be the problem of Puppeteer during fetching? Cheerio code is synchronous, so I don't think Cheerio is the problem. I never get error: cannot get attr('profile') of undefined, so it means there is a header element, but I get error: substring() of undefined. That is why I put a condition before it to check.

const puppeteer = require('puppeteer')
const cheerio = require('cheerio')
const baseUrl = 'https://www.104.com.tw'

const sleep = (milisecond) => {
  return new Promise((resolve, reject) => setTimeout(resolve, milisecond))
}

const scrapeCompanyPage = async (dataList, page) => {
  for (let i = 0; i < dataList.length; i++) {
    await page.goto(dataList[i].companyUrl)
    const html = await page.content()
    const $ = cheerio.load(html)
    const header = $('div.header')
    //sometimes company data below is undefined, but header exists
    dataList[i].companyAddress = header.attr('address') ? header.attr('address') : null
    dataList[i].companyIntro = header.attr('profile') ? header.attr('profile').substring(0, 50) : null 
    await sleep(1000)
  }
  return dataList
}

The website this section of code scrapes is this: https://www.104.com.tw/company/1a2x6bk72b?jobsource=2018indexpoc The content is different for different companyUrl, but the structure is the same.

The below is the HTML tag I want to select.

...

user_15187899 · Answer

Press Ctrl-U and you will see the source code of the main content is empty. The website might be made by React, Vue or other Javascript library to rendering. So you need to wait for the element appear.

But if I inspect the website in Developer Tools > Network tab > XHR Filter, reload the page, you will see the API they call to get these metadata like address, profile, etc... And you might not need to scrape the html.

user_966023 · Answer

The one at that link had a profile and address attribute so that wouldn't happen there.

If the attribute is missing you will get undefined for example $(div).attr('foo')

For node 14+ you can use the optional chaining ? operator to avoid problems with those:

dataList[i].companyIntro = header.attr('profile')?.substring(0, 50)

existing attribute selected by Cheerio sometimes return undefined (get HTML with Puppeteer)

2 Answers