I'm running Puppeteer script both on my Amazon Linux EC2 Instance and my Macbook Air (OSX). The script has to stay in tact at one page and repeatedly perform form-filling tasks over and over again.
I'm encountering issues where i'm running it as a pm2 daemon process, I can see that the memory consumption of that process is increasing every minute or two until it's clogging the server's memory.
In order to overcome it, I used --max-memory-restart flag of pm2 and it prevents the server from crashing, but the node process restarts every 30 minutes in average.
Here is the example that represents the code i'm running:
// Require the puppeteer library
const puppeteer = require('puppeteer');
async function scrape() {
// Create the browser
const browser = await puppeteer.launch({
headless: true,
ignoreHTTPSErrors: true,
args: [
'--no-sandbox',
'--disable-setuid-sandbox',
'--disable-dev-shm-usage',
'--disable-accelerated-2d-canvas',
'--no-first-run',
'--no-zygote',
'--single-process',
'--disable-gpu',
'--ignore-certificate-errors'
]
});
// Wrap scraping/testing code in try
try {
await initialFlow(browser, null); // Page instance is null for the first time
// Catch and log errors
} catch (error) {
// Your chance to handle errors
console.error(error);
} finally {
// Always close the browser
await browser.close();
}
}
const initialFlow = async (browser, page) => { // Save resources by reusing the browser and page instances
try {
if (!page) {
const pages = await browser.pages();
page = pages[0];
}
await page.setRequestInterception(true);
page.on('request', (req) => {
if (req.resourceType() == 'font' || req.resourceType() == 'image') { // Save resources by denying images and fonts from being rendered
req.abort();
} else {
req.continue();
}
});
await page.goto('about:blank'); // Go to blank page
await page.goto('https://www.google.com'); // For the example
await page.waitForSelector('#Form', { visible: true });
return await initialFlow(browser, page); // Perform the actions again on the same browser and page instances
} catch (err) {
return await initialFlow(browser, page);
};
}
// Run our function
scrape().catch(console.error);
When running this example, the memory displayed in the pm2 ls command start from around 30mb and continuously increases as the time goes by.
Any suggestions to prevent this memory leak?