Spacy, using nlp.pipe on large dataset in python, multiprocessing leads to processes going in sleep state. How to proper use all CPU cores?

user_13123141 0
Viewed 1641

I'm working on a NLP classification problem over a large database of emails (~1 million). I need to use spacy to parse texts and I'm using the nlp.pipe() method as nlp.pipe(emails,n_process=CPU_CORES, batch_size=20) to loop over the dataset. The code works but I'm facing a (maybe not so)weird behavior: the processes are being created but they are all in SLEEP state but one, casually some of them go in RUN state for a few seconds and then back to sleep. So I find myself with one single process using one core at 100% but of course the script not using all the CPU cores. It's like the processes don't get "fed" input data from pipe.

Does anybody know how to properly use spacy nlp pipe or anyway how to avoid this situation? no way to use nlp.pipe with the GPU?

Thank you very much! Sandro

EDIT: I still have no solution but i've noticed that if I set the batch_size=divmod(len(emails),CPU_CORES), the processes all starts running at 100% CPU and after a few seconds they all switch to sleeping state but one. It really looks like some element in spacy pipe gets locked while waiting for something to end.... any idea??

EDIT2: Setting batch_size=divmod(len(emails),CPU_CORES) while processing a large dataset leads inevitably to a spacy memory error:

MemoryError: Unable to allocate array with shape (1232821, 288) and data type float32

*thing that is maybe not so weird as my machine has 10GB of RAM and (1232821×288×32)bits / 8 = 1.4GB multiplied by 6 (CPU_CORES) leads to a 8.5GB of RAM needed. Therefore I guess that, having other stuff in memory already, it turns out to be plausible. *

spacy sleep multiprocessing python
2 Answers

I've found that using n_process=n works well for some models, like en_core_web_lg, but fails with others, such as en_core_web_trf.

For whatever reason, en_core_web_trf seems to use all cores with just a batch_size specified, whereas en_core_web_lg uses just one unless n_process=n is given. Likewise, en_core_web_trf fails with a closure error if n_process=n is specified.

user_113490550

Ok, I think I found an improvement but honestly the behavior it's not really clear to me. Now the sleeping processes are way less, with most of them stable running and a few sleeping or switching between the two states. What I've done was to clean and speedup all the code inside the for loop and set the nlp.pipe args like this:

for e in nlp.pipe(emails,n_process=CPU_CORES-1, batch_size=200):

If anybody have any explanation about this or any suggestion on how to improve even more, it's of course more than welcome :)

user_131231410
Related