I'm running a heroku free dyno with 512 MB (I plan to upgrade later) of RAM, on which I'm running a express server to host my api with returns the 'src' attribute of images on the webpage that I want to scrape. Currently my express route looks like this
async function run(){
let browser;
try{
browser = await puppeteer.launch({
args: [
'--no-sandbox',
'--disable-setuid-sandbox',
],
});
const page = await browser.newPage();
await page.setViewport({ width: 1200, height: 1200 });
await page.goto(url,{waitUntil:'load',timeout: 0});
const imgs = await page.$$eval('.img-wrapper img[src]', imgs => imgs.map(img => img.getAttribute('src')));
res.send({imageList:imgs})
}catch(e){
res.send({errormsg:"Something went wrong with the scrapper"});
console.log(e)
}finally{
await browser.close();
}
}
so each time I send a POST request to the express route run() executes.
But the problem is if multiple post request is send (say more than 100) I assume 100 browser instances are opened up and closed (I assume as I used async but I do know JS is synchronous, so I'm not sure), which means I run out of memory and heroku will stop my dyno automatically.
So I was wondering if it is better to declare the browser instance as a global variable and then open and close tab/pages instead, and have the browser always running.
Something like
const browser = await puppeteer.launch({
args: [
'--no-sandbox',
'--disable-setuid-sandbox',
],
});
app.post(....
const page = await browser.newPage();
await page.goto(url,{waitUntil:'load',timeout: 0}); // openpage
//do stuff
finally{
page.close()
})