querying li elements returns NodeList conitaned nothing but should exist

suzu 0
Viewed 37

Thank you for reading. I would appreciate any suggestions or information.

What I'm doing

I'm making web scraping app JSDOM and axios.
Trying to query all of <a href="url"> and get href value.

Problem

  • Why lists 's length is 0?

  • How can I get expected result? I want to get NodeList with 3 nodes

  • Is there any points to be careful about JSDOM? I doubt this is some JSDOM problem.

// target HTML
<a href="#">getLink</a>
// It seems that this a tag is clickable and that gives #download-options "display: none !important; visibility: hidden !important". Does this affect what I'm doing?
<div id="download-options">
    <div class="panel-body">
        ::before
        <ul>
            <li><a href="url1"></a></li>
            <li><a href="url2"></a></li>
            <li><a href="url3"></a></li>
        </ul>
        ::after
    </div>
</div>

// My web-scraping code

let res = await axios.get('url')
conts dom = new JSDOM(res.data)

const ulist = dom.window.document.querySelector('#download-options > .panel-body > ul')
// => returns HTMLUListElement {}
// ulist.childElementCount => returns 1

const lists = ulist.querySelectorAll('li')
// => returns NodeList {}
// lists.length => returns 0 expected 3, so cannot forEach.

NodeList HTMLUlistElement

What I've tried

  • I checked same query code from my google chrome browser developer console, then it returned what I expected. (I got NodeList with 3 nodes and could execute forEach and got all of hrefs value.)

  • added user-agent for axios request.

Thank you for reading. I would appreciate any suggestions or information.

jsdom axios web-scraping javascript html
1 Answers

Turned out that the target site doesn't respond with <li> in <ul> on first request to the site. I don't know why and how the site does this, but I think its related to cookies or cache things.

So I used puppeteer to visit the site's homepage and then target page. This solved the problem.

Codes are like below

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch({headless: true});
  const page = await browser.newPage();
  await page.goto('http://example.com') //go to homepage to solve cache? problem
  await page.goto('https://example.com/targetpage'); // then go to actual target
  await page.waitForSelector('#download-options li'); // wait for it just in case

  const ul = await page.$("#download-options ul") 
  const lis = await ul.$$("li")

  for await (const li of lis ) {
    const a = await li.$('a')
    const hrefValue = await a.evaluate((el) => el.getAttribute('href'))
    console.log(hrefValue)
  }
  await browser.close();
})
suzu0
Related