Puppeteer Page won't load after .click() until reaching the end of the block

Viewed 87

I'm using puppeteer to load a page, click on an button that loads a different page, send a Telegram message which the user will use to decide what to scrape in the new page (I'm using Telegraf), and then sleep for some time. My block looks like this:

{  
  await page.goto(url)
  await page.click(selector)
  await sendTelegramMessage(page)
  await sleep(ms)
}

I also added the option to skip the telegram message inside it's function like this:

async function sendTelegramMessage(){
  if (telegramEnabled){
    // Send message
    // Call another function to handle user response. The code scrapes content based on response.
    // Page gets passed to the handler and the handler passes it to the scraper.
  } else {
    // Assume user response and call the scraper directly.
    // Page gets passed to the scraper.
  }
}

If the telegram message is disabled, sendTelegramMessage() calls the scraper and passes the page. The scraper returns error.

After some debugging with non headless Chromium I realized that the problem is as follows:

  1. After .click() the new page starts loading
  2. sendTelegramMessage() calls the scraper but the page hasn't finished loading so it throws an error

Debugging with telegramEnabled = true:

  1. After .click() the new page starts loading
  2. sendTelegramMessage() sends the message and calls the response handler, which waits for the user to interact with its chat to call the scraper
  3. sendTelegramMessage() finishes and the main block continues to the sleep function. At this point page is still loading. Finishes loading when sleep() gets called.
  4. User interacts with telegram chat and the handler passes page to the scraper, which works fine.

The way I see it, when a button is clicked, its link won't load until code reaches the end of the block or at least the sendTelegramMessage() finishes. If telegramEnabled is false the function will call the scraper which will return error since page didn't load and code will never reach the end of the block, so the loading will never end. Non headless Chromium just shows a page loading forever. When telegramEnabled is true, the code reaches the end of the block so the page finishes loading and the scraper will never throw an error.

Do you have any idea why this happens? For some reason clicking on a link on the page doesn´t load immediately, it just waits for some code to be executed.

0 Answers
Related