Puppeteer: memory leak error and timeout handling

user_1629932 0
Viewed 1187

I am trying to scrape the data from this site. Below is the code, which is working for single district and hospital type but break when I put it in the loop. I tried putting the statements inside getData function to be async await but it still opens hundreds of browser instances leading to memory leak problems. How do I fix it so that it runs one at a time instead of all at once.

const puppeteer = require('puppeteer');
const districtList = ["020", "001", "003", "008", "016", "017", "004", "006", "005", "007", "022", "021", "019", "009",
  "012", "013", "023", "010", "002", "011", "014", "015", "018"];
outputJsonArray = [];
const url = "https://excise.wb.gov.in/CHMS/Public/Page/CHMS_Public_Hospital_Bed_Availability.aspx";

async function scrape(did, hospType) {
  // const browser = await puppeteer.launch({headless: false, args: ['--auto-open-devtools-for-tabs']});
  const browser = await puppeteer.launch({headless: false});
  const page = await browser.newPage();
  page.setDefaultNavigationTimeout(0);
  await page.goto(url, {waitUntil: 'networkidle0', timeout: 0});
  await page.waitForSelector('#ctl00_ContentPlaceHolder1_ddl_District');
  await page.click(hospType);
  await page.select('#ctl00_ContentPlaceHolder1_ddl_District', did);
  await page.waitForSelector('tbody tr');
  // extracting information from code
  let outputData = await page.evaluate(() => {
    let output = [];
    let dataRows = document.body.querySelectorAll("tbody tr");
    let cityName = document.querySelector("#ctl00_ContentPlaceHolder1_ddl_District").selectedOptions[0].textContent.trim();
    console.log(cityName)
    // return dataRows.length;
    console.log("Num entries == " + dataRows.length);
    dataRows.forEach((dataRow) => {
      let rowJson = {};
      // debugger;
      rowJson["Name"] = dataRow.querySelector('h5').textContent.trim().replace(/\s+/g, ' ');
      // let h3 = dataRow.querySelectorAll('li h3.text-success')[3].textContent;
      rowJson["Contact"] = dataRow.querySelector('.rounded-pill.bg-success').textContent.trim().replace(/\s+/g, ' ');      
      output.push(rowJson);
    })
    return output;
  });
  outputJsonArray = outputJsonArray.concat(outputData);
  console.log("data == ");
  console.log(outputData);
  await browser.close();
};

districtList.forEach(getData);
async function getData(districtId) {
  await scrape(districtId, "[for=ctl00_ContentPlaceHolder1_rdo_Govt_Flag_0]")
  await scrape(districtId, "[for=ctl00_ContentPlaceHolder1_rdo_Govt_Flag_1]")
  await scrape(districtId, "[for=ctl00_ContentPlaceHolder1_rdo_Govt_Flag_2]")
  console.log("+++++++++++++++++Full JSON below+++++++++++++++++");
  console.log(outputJsonArray);
}

Further, at line 17, waitForSelector timeouts and halts the execution completely. This is because there are some cases for which no results are available. For example, selecting "ALIPURDUAR" in district and 2nd hospital type does not return any results, therefore it keeps waiting and is supposed to timeout. How do I handle these two issues?

(node:50036) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 exit listeners added to [process]. Use emitter.setMaxListeners() to increase limit (Use node --trace-warnings ... to show where the warning was created) (node:50036) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 SIGINT listeners added to [process]. Use emitter.setMaxListeners() to increase limit (node:50036) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 SIGTERM listeners added to [process]. Use emitter.setMaxListeners() to increase limit (node:50036) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 SIGHUP listeners added to [process]. Use emitter.setMaxListeners() to increase limit

puppeteer node.js javascript
2 Answers

You're probably getting errors in the browser context. Try optional chaining and see if that helps:

let data = await page.$$eval('tbody tr', trs => trs.map(tr => {
  return {
    name: tr.querySelector('h5')?.textContent?.trim()?.replace(/\s+/g, ' '),
    contact: tr.querySelector('.rounded-pill.bg-uccess')?.textContent?.trim().replace(/\s+/g, ' ')
  }
}))

EDIT

So to run these one at a time you might do something like:

async function run(districtList){
  for(let districtId of districtList){
    await getData(districtId)
  }
}

run(districtList)

The problem was with forEach which looks like a for loop but isn't.

user_9660230

I would recommend more reliable solution: for every district and hospital run a child process which imports puppeteer. In this way main process will never leak, and each child process might leak but you don't care because you kill it ASAP and it releases all resources. I know it is not documented but it solves the issue very well.

Another links:

user_34791250
Related