Is there any way to wait for dynamic content to be added after page.click() in puppeteer

user_6720428 0
Viewed 1627

I have been writing a code which scrapes the Trailblazer site. The website is a dynamic one, So when i click the Show More button it loads the other badges owned the user. So is there a way for me to click that button and then wait for the new content to be added before loading the markup in cheerio for scraping.

Currently i am just making the broswer wait for 3 seconds before re-click but it makes the whole process take a lot of time and it may also fail if the browser fails to fetch in that time.

So is there any alternative to this? Please find my code below -->

    const url = "https://trailblazer.me/id/akganesa";

  // function to load the page
    async function getPage() {
    const browser = await puppeteer.launch({headless: true});
    const page = await browser.newPage();
    await page.goto(url, {waitUntil: 'networkidle0'});

    const [first_button] = await page.$x("//button[contains(., 'Show More')]");
    await first_button.click();
    while (
      (await (await page.$x("//button[contains(., 'Show More')]")).length) > 0
    ) {
      const [button] = await page.$x("//button[contains(., 'Show More')]");
      await button.click();
      await page.waitForResponse(response => response.status() === 200);
    }

    const html = await page.content(); // serialized HTML of page DOM.
    await browser.close();
    return html;
  }

      // using cheerio to scrape
      const html = await getPage();
      const $ = cheerio.load(html);
cheerio puppeteer node.js
1 Answers

The code below works for what you're asking. The reason I have it click [first_button] before entering the while loop is that the first "show more" button doesn't actually make a network request. 

const go = async () => {
  const browser = await puppeteer.launch({
    headless: false,
    args: [
      "--no-sandbox",
      "--disable-setuid-sandbox",
      "--window-size=1600,1200"
    ],
    defaultViewport: null
  });
  const context = await browser.createIncognitoBrowserContext();
  const page = await context.newPage();
  try {
    await page.goto("https://trailblazer.me/id/akganesa", {
      waitUntil: "networkidle2"
    });
    const [first_button] = await page.$x("//button[contains(., 'Show More')]");
    await first_button.click();
    while (
      (await (await page.$x("//button[contains(., 'Show More')]")).length) > 0
    ) {
      const [button] = await page.$x("//button[contains(., 'Show More')]");
      await button.click();
      await page.waitForResponse(response => response.status() === 200);
    }
    browser.close();
    return;
  } catch (err) {
    console.log(err);
    browser.close();
    return;
  }
};

go();
user_120202950
Related