Puppeteer Memory Increase Problems

Viewed 530

I'm running Puppeteer script both on my Amazon Linux EC2 Instance and my Macbook Air (OSX). The script has to stay in tact at one page and repeatedly perform form-filling tasks over and over again.

I'm encountering issues where i'm running it as a pm2 daemon process, I can see that the memory consumption of that process is increasing every minute or two until it's clogging the server's memory.

In order to overcome it, I used --max-memory-restart flag of pm2 and it prevents the server from crashing, but the node process restarts every 30 minutes in average.

Here is the example that represents the code i'm running:

// Require the puppeteer library
const puppeteer = require('puppeteer');

async function scrape() {
    // Create the browser
    const browser = await puppeteer.launch({
        headless: true,
        ignoreHTTPSErrors: true,
        args: [
            '--no-sandbox',
            '--disable-setuid-sandbox',
            '--disable-dev-shm-usage',
            '--disable-accelerated-2d-canvas',
            '--no-first-run',
            '--no-zygote',
            '--single-process',
            '--disable-gpu',
            '--ignore-certificate-errors'
        ]
    });

    // Wrap scraping/testing code in try
    try {
        await initialFlow(browser, null); // Page instance is null for the first time
        // Catch and log errors
    } catch (error) {
        // Your chance to handle errors
        console.error(error);
    } finally {
        // Always close the browser
        await browser.close();
    }
}

const initialFlow = async (browser, page) => { // Save resources by reusing the browser and page instances
    try {

        if (!page) {
            const pages = await browser.pages();
            page = pages[0];
        }

        await page.setRequestInterception(true);

        page.on('request', (req) => {
            if (req.resourceType() == 'font' || req.resourceType() == 'image') { // Save resources by denying images and fonts from being rendered
                req.abort();
            } else {
                req.continue();
            }
        });

        await page.goto('about:blank'); // Go to blank page
        await page.goto('https://www.google.com'); // For the example

        await page.waitForSelector('#Form', { visible: true });
        return await initialFlow(browser, page); // Perform the actions again on the same browser and page instances

    } catch (err) {
        return await initialFlow(browser, page);
    };
}

// Run our function
scrape().catch(console.error);

When running this example, the memory displayed in the pm2 ls command start from around 30mb and continuously increases as the time goes by.

Any suggestions to prevent this memory leak?

0 Answers
Related