Protocol error (Runtime.callFunctionOn): Target closed

Viewed 1292

I want to scrape a page with some news inside. Here it's an HTML simplified version of what I have :

<info id="random_number" class="news"> 
    <div class="author">
        Name of author  
    </div>
    <div class="news-body">
        <blockquote>...<blockquote>
        Here it's the news text
    </div>
</info>
<info id="random_number" class="news"> 
    <div class="author">
        Name of author  
    </div>
    <div class="news-body">
        Here it's the news text
    </div>
</info>

I'm getting the author and text body of each news, without the blockquote part.

So I wrote this code :

async function main() {
    const browser = await puppeteer.launch({args: ['--no-sandbox', '--disable-setuid-sandbox']});
    const page = await browser.newPage();
    await configPage.goto('www.url.com', {waitUntil: 'networkidle2'});
    while (true) {
          await page.reload({ waitUntil: ["networkidle2"] });
          let newsPage = await page.$$("info.news");
          for (var news of newsPage){ // Loop through each element
             let author = await comment.$eval('.author', s => s.textContent.trim());
             let textBody = await comment.$eval('.news-body', s => { s.querySelector('blockquote')?.remove(); return s.textContent.trim(); });

             console.log('Author :'+ author);
             console.log('TextBody :'+ textBody);
          }
          ...Wait some time
    }

}

main();

It works well, but sometimes, after a few times, I have this : Error: Protocol error (Runtime.callFunctionOn): Target closed.

On the line :

let author = await comment.$eval('.author', s => s.textContent.trim());

How handle this ?

Thank you

0 Answers
Related