scroll to bottom not applying to site puppeteer

seriously 0
Viewed 50

I am trying to use puppeteer to try and scroll all the way to the bottom of the site but the code I am using is not working. What I did was set a while loop then check if new height equals previous height then set a promise but for some reason it is not working. Where did I go wrong and how can I fix it. Thanks in advance.

const puppeteer = require('puppeteer');

const scrapeInfiniteScrollItems = async(page) => {
  while (true) {
    previousHeight = await page.evaluate('document.body.scrollHeight')
    await page.evaluate('window.scrollTo(0, document.body.scrollHeight)')
    await page.waitForFunction(`document.body.scrollHeight > ${previousHeight}`)
    await new Promise((resolve) => setTimeout(resolve, 1000));
  }
}


(async() => {
  const browser = await puppeteer.launch({
    headless: false
  });
  const page = await browser.newPage();
  await page.goto('https://www.youtube.com', {
    waitUntil: 'networkidle2',
  });

  await scrapeInfiniteScrollItems(page)
})();

puppeteer javascript
1 Answers

In case of youtube the height of body is 0 that's why your function is not working. If we see in devtools on youtube the whole content is in ytd-app element.

So we should use document.querySelector('ytd-app').scrollHeight instead of document.body.scrollHeight to scroll down to bottom.

working code.

const scrapeInfiniteScrollItems = async (page: puppeteer.Page) => {
  while (true) {
    const previousHeight = await page.evaluate(
      "document.querySelector('ytd-app').scrollHeight"
    );
    await page.evaluate(() => {
      const youtubeScrollHeight =
        document.querySelector("ytd-app").scrollHeight;
      window.scrollTo(0, youtubeScrollHeight);
    });
    try {
      await page.waitForFunction(
        `document.querySelector('ytd-app')?.scrollHeight > ${previousHeight}`,
        { timeout: 5000 }
      );
    } catch {
      console.log("done");
      break;
    }
    await new Promise((resolve) => setTimeout(resolve, 1000));
  }
};
Kashif Ghafoor0
Related