puppeteer consuming too much disk space with temporary files

user_14459185 0
Viewed 1333

I'm having problems with the puppeter, I need to optimize its performance and also know how to get it to pay for the temp files.

the arguments I use are

 args: [
                    `--window-size=800,600`,
                    '--no-sandbox',
                    '--disable-setuid-sandbox',
                    '--disable-dev-shm-usage',
                    '--disable-accelerated-2d-canvas',
                    '--no-first-run',
                    '--no-zygote',
                    '--disable-gpu'
                ]

Would there be any way to optimize it and not make it consume a lot of space on my disk with temporary files?

I open it more than 1000x a day with the scraper I make.

there's no way to be in headles

puppeteer node.js
1 Answers

Disabling the cache should reduce most if not all of your temporary files. Add the following args, like so:

'--aggressive-cache-discard',
'--disable-cache',
'--disable-application-cache',
'--disable-offline-load-stale-cache',
'--disable-gpu-shader-disk-cache',
'--media-cache-size=0',
'--disk-cache-size=0',

Moreover, you may want to disable images and CSS (see here) to further increase the speed and decrease temporary files and network throughput.

Also, there are additional flags that may be helpful for your use case:

'--disable-extensions',
'--disable-component-extensions-with-background-pages',
'--disable-default-apps',
'--mute-audio',
'--no-default-browser-check',
'--autoplay-policy=user-gesture-required',
'--disable-background-timer-throttling',
'--disable-backgrounding-occluded-windows',
'--disable-notifications',
'--disable-background-networking',
'--disable-breakpad',
'--disable-component-update',
'--disable-domain-reliability',
'--disable-sync',

If you try all that and still find that it generates a lot of garbage, you may want to look into Dockerizing your scraper. I recommend docker-slim if you do go the Docker path, as it significantly reduces the size of Docker images without any negative side effects.

user_114012010
Related