I'm parallelizing a Python scraping task using ChromeDriver on a multicore (16) linux machine.
My .py creates multiple threads, and each thread loops forever, each cycle launching and destroying a ChromeDriver instance. On my 16-core box I might set NTHREADS_MAX = 100, and most threads will be encasing a running ChromeDriver at any given moment.
I wish to maximize concurrency. But as RAM/CPU usage gets close to 100% the system starts to chugg, naturally. So I need some kind of throttling mechanism.
I can see that each ChromeDriver is being run on a randomly assigned core.
My question is: What's a sensible architecture?
I'm thinking one core should be reserved for the 'controller' .py process. But then how to ensure a ChromeDriver doesn't attempt to run on the same core?
AFAICS that gets messy fast. I know there's a technique to manually force a running process onto a specific core. But I can't see any way to SPAWN it on a specific core, or to reserve a particular core for the parent .py process.
Also I'm struggling to get traction on the throttling task. There's RAM and CPU to consider. I'm considering putting while availRAM < 1GB or availCPU < 10%: sleep(0.1) in front of any heavy task (like starting up a ChromeDriver instance or loading a webpage).