Puppeteer page.goTo() never finishes

Question

This is another example of this happening ^

I've never understood why, but even the simplest of puppeteer scripts fails to get passed goTo()

I have the following code:

    const browser = await puppeteer.launch({ executablePath: '/usr/bin/google-chrome-unstable', args: ["--proxy-server='direct://'", '--proxy-bypass-list=*', '--no-sandbox', '--disable-setuid-sandbox'] });
    const page = await browser.newPage();
    await page.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36');
    await page.setDefaultNavigationTimeout(0);

    log('before nav');
    await page.goto('http://www.google.com');

    log('waiting nav');
    await page.waitForNavigation({
        waitUntil: 'networkidle0'
    });

    log('complete');

    await browser.close();

output is:

$:# node lib/tests/nav.test.js
    before nav
    waiting nav

It never ever logs "complete". I have tried without the proxy-bypass and proxy-server, no-sandbox on and off. I have tried networkidle2. It doesn't complete. I also tried various different websites. I am on Puppeteer 4.x and using node 12.x and npm 6.x.

I have investigated the Promise method that people suggest:

await Promise.All([ page.goto('http://www.google.com'), page.waitForNavigation() ]);

and this works sometimes. but it's very sporadic.

Is puppeteer really this buggy? Is there ANY known way to guarantee a page load before performing actions. I need to nav around and fill in forms and click buttons etc so the elements all need to be there.

I also tried the waitForSelector() version too, that doesn't load either.

I feel like puppeteer is fundamentally broken. Any ideas?

user_10401931 · Answer

You're not using Puppeteer the way you should, that's why it feels buggy, but it's mostly because you don't know what's going on in the script.

Let's have a look.

await page.goto('http://www.google.com');

this really means this:

await page.goto('http://www.google.com', { waitUntil: 'load' });

So you're going to a page and waiting till the load event fires. That means that when you reach behind this line, there's no other navigation to wait for. Therefore this line:

await page.waitForNavigation();

will wait forever. That's what you describe as "it hangs".

So since the method page.goTo() already offers ways to specify when the navigation succeeded (currently they are: load, domcontentloaded, networkidle0, networkidle2), you don't really need to use any other method in combination with page.goTo().

Another situation is when you click a button and expect something to happen, like a navigation or a selector to appear in page. Since the method page.click() doesn't offer the same options as page.goTo(), you often need to use another method in combination, like so:

await Promise.all([
    page.waitForNavigation(),
    page.click(selector)
]);

or to wait for navigation and some selector:

await Promise.all([
    page.waitForNavigation({ waitUntil: 'networkidle0' }),
    page.waitForSelector(selector),
    fbButton.click()
]);

If you stick to these patterns, I don't think you'll feel Puppeteer is buggy. It's just it's a low level tool, so you need to write more code than in some other framework like WebdriverIO and the like.

Puppeteer page.goTo() never finishes

1 Answers