Why puppeteer page.goto() hangs?

user_4845952 0
Viewed 1707

I'm running puppeteer in an alpine-chrome container with sandboxing, everything is done exactly as the docs suggest. And I've created a whole bunch of different puppeteer-fueled servers.

I'm launching Puppeteer without any arguments, except headless: false.

Why is await page.goto() method so problematic? It keeps giving me all kinds of non-sensical problems.

I'm always doing this:

await page.goto('https://my-url.com/something', {
  waitUntil: 'networkidle2',
  timeout: 0
})

Right now it just randomly hangs. Nothing happens, code execution just stops there. All I see is that CPU utilization hits the roof and then down to slightly-elevated levels, comparing to the properly-ongoing execution. 90% of time the goto doesn't elevate CPU utilization at all. Same page, same server, hangs randomly within the same script execution.

Why is this hapening? Do I have to put setTimeout and try-catch statements all over my code around each page.goto() method? It feels like I'm using puppeteer fundamentally incorrect, because this behaviour doesn't make sense.

Sorry for vague question, I don't understand what kind of information would be helpful here.

google-chrome-headless puppeteer
2 Answers

It looks like the goto method is throwing you errors. Therefore it's important to review this tidbit of info from puppeteer documentation -

https://devdocs.io/puppeteer/index#pagegotourl-options

page.goto will throw an error if:

-there's an SSL error (e.g. in case of self-signed certificates).

-target URL is invalid. the timeout is exceeded during navigation.

-the remote server does not respond or is unreachable.

-the main resource failed to load.

I would also recommend trying networkidle0 and maybe not passing a timeout parameter. It's odd you're having this issue as normally its not page.goto() function giving me issues but working with the selectors.

user_118105100

It hangs because you have disabled timeout by setting timeout: 0 option and it is waiting until there are no more than 2 network connections for at least 500 ms. You can set timeout option in milliseconds. The following will throw timeout error if page.goto() is not completed within 30 seconds.

await page.goto('https://my-url.com/something', {
  waitUntil: 'networkidle2',
  timeout: 30000,
})

Please refer to puppeteer documentation at https://devdocs.io/puppeteer/index#pagegotourl-options .

user_131194890
Related