How do I visit one url after another using puppeteer

user_8182731 0
Viewed 758

I have started learning puppeteer and trying to execute some code to improve my learning. I would like to achieve simple task visit one URL after another However I m not able to execute it. request you to please review my code and suggest any changes.

 const puppeteer = require('puppeteer');
    (async () => {
        const browser = await puppeteer.launch({ headless: false });
        const page = await browser.newPage();
        const urls = ['https://www.google.com/', 'https://www.yahoo.com/']
        for (let i = 0; i < urls.length; i++) {
            const url = urls[i];
            await page.goto(`${url}`);
            await page.waitForNavigation({ waitUntil: 'networkidle2' });
        }
        await browser.close();
    })();

above code only visit google.com and then throws below error .

node .\testAutomation.js
(node:8564) UnhandledPromiseRejectionWarning: TimeoutError: Navigation timeout of 30000 ms exceeded
    at D:\Document\study\jswebscrapping\node_modules\puppeteer\lib\cjs\puppeteer\common\LifecycleWatcher.js:106:111
    at async FrameManager.waitForFrameNavigation (D:\Document\study\jswebscrapping\node_modules\puppeteer\lib\cjs\puppeteer\common\FrameManager.js:125:23)
    at async Frame.waitForNavigation (D:\Document\study\jswebscrapping\node_modules\puppeteer\lib\cjs\puppeteer\common\FrameManager.js:429:16)
    at async Page.waitForNavigation (D:\Document\study\jswebscrapping\node_modules\puppeteer\lib\cjs\puppeteer\common\Page.js:836:16)
    at async D:\Document\study\jswebscrapping\testAutomation.js:16:9
(node:8564) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:8564) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
puppeteer javascript
1 Answers

After await page.goto();, you do not need await page.waitForNavigation();. The former command already includes awaiting for the navigation, so the latter just hangs after it.

await page.waitForNavigation(); is needed after some user interaction (e.g. a click) or script command that change a URL.

So, just delete the await page.waitForNavigation({ waitUntil: 'networkidle2' });.

user_26255600
Related