100,000 page renders using headless browser, max per cpu per second on aws

user_3866246 0
Viewed 563

I'm doing a scraping project and need to use a browser to load javascript on websites that I visit.

If using just http I can get up to 150/second on an m4.large. Which has 2 cpus. On initial tests using headless chrome its so much slower around 4 / second and typically the bottleneck is CPU not RAM. For many web pages basic http is fine but increasingly its not with single page apps etc.

Does anyone have some experience doing something similar? What were you able to achieve? What was the stack you used e.g. python, selenium, headless chrome and multiprocessing. Did you have to alter the configuration of chrome?

web-scraping amazon-web-services selenium chromium python
1 Answers

This particular page on Stack Overflow takes 1.2 seconds to load on my machine, according to the "Performance" tab in Chrome's devtools:

Performance tab

You can break that down by CPU usage or wall clock time, but the point is that it takes a while.

In this case I would load the page once in a browser to learn what resources are accessed, then execute those using a load testing tool other than a browser.

Copy All as cURL

Naively executing this curl command — first parallelizing it by replacing ; with & — takes 1.8 seconds on my machine. But the point is that you only want to hit the server, not interpret any HTML, CSS, JavaScript, or images.

But if you want to crawl the web and see what really happens upon each page load, so that you can look at the page and decide where to go next, there isn't much to do besides a headless browser. Real production web crawlers have a much larger machine budget, and possibly use clever techniques to avoid wasting CPU on decoding images and laying out CSS.

user_197500
Related