Puppeteer CPU 100% server kills the process

user_2880383 0
Viewed 187

I need to run as many browsers as we can but once CPU goes to 100% for some secs aws server is killing the scrapper process. Is there a way to limit cpu use to maybe 95 so the process does not get removed?

params I tried:

const options = {
    args: [
      '--no-sandbox',
      '--disable-setuid-sandbox',
      '--disable-dev-shm-usage',
      '--disable-accelerated-2d-canvas',
      '--no-first-run',
      '--no-zygote',
      '--single-process', // <- this one doesn't works in Windows
      '--disable-gpu'
    ],
    headless: true
  }

  return await puppeteer.launch(options)

I need to be able to launch N browsers to scrape but never hit the 100%.

puppeteer node.js amazon-ec2
1 Answers

Typically when a server CPU is floored at 100% usage, you will experience lockups, very slow processes, time-outs, and just unpredictable things. It isn't unheard-of for servers to be running at 100% CPU usage--as some software and programs out there have been designed to run at that percentage.

There probably is some AWS tiers out there where they send out e-mails trying to ask to restart your machine if the CPU has been 100% for X amount of hours. Hopefully that isn't happening with your plan/service.

When a server is OOM (out-of-manamemory) or RAM is at 100%, that is when the OS will start killing processes. The order in which it may kill the process is usually based off a priority, execution time, or a secret algorithm no one could understand.

  1. There is not going to be a way to limit CPU use to an arbitrary number below 100% using the puppeteer.launch() options/args. At least not as far as I have ever seen.

  2. Test your code for performance problems. There are several ways to troubleshoot your program/code to ensure that it isn't your own program that is causing the problem. Puppeteer + JavaScript in general makes it easy for a memory leak to occur which could be a huge issue. Multiple large loops over a Promise with an Array of ElementHandles (page.$$) can be a huge performance killer.

  3. Failing to close a fs (file stream) or browsers & pages can add up eventually too. Lucky for us, we have Garbage Collection to come and try to fix the problem; while at the same time using more memory/CPU.

  4. Use Kubernets with a Puppeteer Manager to manually control your own browsers. I highly recommend taking a look at the Digital Ocean Tutorial | Puppeteer + node.js + Kubernetes + Docker + Load Balancing about using a variety of tools to make scraping a very organized and abstract process. Assuming you haven't already seen it or have used it.

  5. Kubernetes + Docker + Load Balancing can all be done on AWS too. Adding in tools like Kubernetes + Docker brings huge functionality for monitoring resources. An example would be: pausing a container and restarting it on a different balancer. Plus many other things that are too specific for this answer.

  6. Lastly, I think the best way to keep your browsers stable is to monitor your server and identify how many browsers you can run before it starts getting towards the OOM / 100% RAM usage. Not worry as much about the 100% CPU limit, unless it is significantly impacting performance.

Hope this helps or adds some insight into your question.
user_17123450
Related