Using Promise.all to achieve sort of multithreading with Puppeteer

user_10760405 0
Viewed 359

I would like to consult may way of thinking.

Writing scraping bots or bots that perform certain activities on websites (using Puppeteer), I often need 'sort of' multithreading functionality, to be able to approach a number of pages at the same time and perform certain actions on them, preferably at the same time as well.

For this purpose, I use Promise.all() following this pattern:

const runInParallel = async(len) => {

    // create an array with a number of elements corresponding to required number of functions
    // to be performed at the same time 
 
    // these can also be URLs if I happen to know them beforehand 

    const iterations = [...Array(len).keys()]; 

    // create an array of promises that run in parallel 
    return await Promise.all(
        iterations.map(async i => {
            try {
                // use puppeteer to access a page, get data or perform certain actions 
                await scrape(); 
            } catch (e) {
                // handle error 
            } finally {
                // close page and browser
            }
        })
    ); 
}

Normally, the above is wrapped in yet another loop where each iteration awaits all the promises to be resolved/rejected before it starts the next iteration, this way i can access a number of pages at the same time, await all the actions to be completed on all of them and move to the next iteration where the process is repeated.

I am wondering what are the drawbacks of this approach and if there is a better alternative for the purpose of accessing a number of pages at the same time to either scrape data or perform certain actions on them.

puppeteer async-await multithreading javascript
1 Answers

This approach works, and it does real multithreading not just "sort of". Puppeteer launches Chromium in separate processes and they all run concurrently.

Watch out for memory issues though. I experimented with a similar strategy and I found that starting 2-3 browsers consume all the memory in my machine and everything grinds to a halt. You might want to limit how many scraping jobs you do in parallel. Here is an article I wrote how to solve this.

You can also optimize this workflow by using only 1 browser instance but different pages. That way, the scrape call can open a new tab, do the scraping, then close it, and multiple scrape jobs can share the same browser.

user_20321540
Related