I want to scrape a page with some news inside. Here it's an HTML simplified version of what I have :
<info id="random_number" class="news">
<div class="author">
Name of author
</div>
<div class="news-body">
<blockquote>...<blockquote>
Here it's the news text
</div>
</info>
<info id="random_number" class="news">
<div class="author">
Name of author
</div>
<div class="news-body">
Here it's the news text
</div>
</info>
I'm getting the author and text body of each news, without the blockquote part.
So I wrote this code :
async function main() {
const browser = await puppeteer.launch({args: ['--no-sandbox', '--disable-setuid-sandbox']});
const page = await browser.newPage();
await configPage.goto('www.url.com', {waitUntil: 'networkidle2'});
while (true) {
await page.reload({ waitUntil: ["networkidle2"] });
let newsPage = await page.$$("info.news");
for (var news of newsPage){ // Loop through each element
let author = await comment.$eval('.author', s => s.textContent.trim());
let textBody = await comment.$eval('.news-body', s => { s.querySelector('blockquote')?.remove(); return s.textContent.trim(); });
console.log('Author :'+ author);
console.log('TextBody :'+ textBody);
}
...Wait some time
}
}
main();
It works well, but sometimes, after a few times, I have this : Error: Protocol error (Runtime.callFunctionOn): Target closed.
On the line :
let author = await comment.$eval('.author', s => s.textContent.trim());
How handle this ?
Thank you