Multiprocess Pool vs asyncio.run_in_executor

Viewed 1697

Overview

I am trying to parallelize a text classification project that certainly takes too long to run completely in series. I've attempted two possible variations of this that I believe function similarly and am curious about the results of each that I've seen in the Resource Monitor.

The First Solution

The first thing that I tried was to use Python's asyncio with run_in_executor(). This is a method I had used previously to parallelize some simple jobs, so I thought I'd give it a try. It looks like this:

cores = cpu_count()
qty = len(data)
last = 0

coros = []
loop = asyncio.get_event_loop()
for i in range(cores):
    top = ceil((qty * (i + 1)) / cores)
    cor = loop.run_in_executor(None, vect.transform, data[last: top])
    last = int(top)

vectors = loop.run_until_complete(asyncio.gather(*coros))
loop.close()

My understanding is that this code creates a default pool of threads, runs func on each chunk of the data and returns the results in the same order they came in. I expected this to top out my cpu but it only ever seems to run at 15%. Looking at the individual processors, only half of them are engaged and only at around 20%-30% each. This remains even until I get too impatient and stop the program or try to change something to streamline it.

The Second Solution

The second method I'm employing utilizes the multiprocessing module and its Pool().map(). This was new to me, but it appeared to function similarly to the first solution but using processes, so I thought it might utilize the processors in a manner closer to what I was expecting. That solution looks like this:

cores = cpu_count()
qty = len(data)
last = 0

chunks = []

for i in range(cores):
    top = ceil((qty * (i + 1)) / cores)

    chunks.append(data[last: top])
    last = int(top)

with Pool(processes=cores) as pool:
    vectors = pool.map(vect.transform, chunks)

I believe that this code should chunk the data, then open new processes to handle the data. Initially, the Resource Monitor does exactly as I would expect. CPU Usage goes to 100% and all cores show fully active. About 15s later, most of the cores are parked and two or three of them at around 15%. Several minutes after that, each core seems to fill up in pockets here and there averaging around 45% at any given time and eventually finishes.

My expectation/The Question

Later in the program, I'm fitting the data to an sklearn classifier that has n_jobs=-1. When this happens all of the cores go 100%. This is what I would like to do. So I have a couple questions:

  1. What is sklearn doing that I am not?
  2. Is there something I'm overlooking in my approach that makes my expectations unrealistic?
1 Answers
Related