I am trying to scrape github contributor insights using NodeJS, puppeteer, and cheerio.
const cheerio = require('cheerio');
const puppeteer = require('puppeteer');
const url = 'https://github.com/grey-software/grey.software/graphs/contributors';
(async () => {
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.waitForSelector('#contributors', {
visible: true,
})
await page.goto(url);
const pageContent = await page.content()
const $ = cheerio.load(pageContent);
$('.contrib-person').each(function (i, elem) {
console.log(elem)
});
await browser.close();
})();
I get the following error when I run the code above
UnhandledPromiseRejectionWarning: TimeoutError: waiting for selector "#contributors" failed: timeout 30000ms exceeded
The #contributors div should load within the 30 seconds, but I always get this timeout. Note: page.waitForNavigation() gives the same timeout error