Run Puppeteer in a loop without refreshing

user_11052638 0
Viewed 372

I am trying to read a website that is updated continuously without the need for refreshing. When I run a Puppeteer scrip to grab the data, it will refresh the page which is not what I want. How would I read the same site without refreshing?

Here is what I currently have:

const puppeteer = require('puppeteer');

(async function test() => {
  const browser = await puppeteer.launch({
    args: ['--no-sandbox', '--disable-setuid-sandbox'],
    executablePath: '/usr/bin/chromium-browser'
  });
  const page = await browser.newPage();
  await page.goto('url-goes-here');

  const data = await page.evaluate(() => {
      let headingFromWeb = document.querySelectorAll("span[class=messageBody]");
      const headingList = [...headingFromWeb];
      return headingList.map(h => h.innerHTML);
  });
  
  console.log(data);
  
  await browser.close();
  setTimeout(test, 1000);
})();

When I try to remove the await browser.close(); and put the 'grabbing data' part in a function with its own setTimeout I get an asynchronous function error.

Any help would be appreciated, thanks!

puppeteer node.js
1 Answers

Maybe something like this?

while (true) {
  await page.waitForTimeout(1000);
  const data = await page.evaluate(() => {
    // ...
  });
  console.log(data);
}

Also, you can consider another approach: running the function on element changes, see Puppeteer - update content of a scraped website for example.

user_26255600
Related