Why Pandas creates mutiple threads while its internal operation is single threaded?

user_825860 0
Viewed 72

If my was not mistaken, Pandas is single threaded in its internal operations. However, I noticed today that running a simple program as below will result as many threads as the number of CPU cores avaiable in the system being created. Why it will create these extra threads?

import threading
import pandas as pd

def use_some_cpu(row):
    print(f'thread id={threading.get_ident()}')
    x = 1.001
    for i in range(100000):
        x *= 1.001

df = pd.DataFrame(list(range(0, 10000)), columns=['foo'])
df.apply(use_some_cpu, axis=1)

If you try to run the program, you will see all the thred id values printed out are the same, which means the actual processing is done from a single thread. However, use htop command, you will see many threads are created by the program (as many as the number of cores in your system) and only one core is busy.

The test is done on Ubuntu 18.04 with pandas 1.0.2 and python 3.7.

pandas multithreading
1 Answers

I can't reproduce this with modern pandas:

In [2]: import threading
   ...: import pandas as pd
   ...: 
   ...: thread_ids = set()
   ...: 
   ...: def use_some_cpu(row):
   ...:     thread_ids.add(threading.get_ident())
   ...:     x = 1.001
   ...:     for i in range(100000):
   ...:         x *= 1.001
   ...: 
   ...: df = pd.DataFrame(list(range(0, 10000)), columns=['foo'])
   ...: df.apply(use_some_cpu, axis=1)
Out[2]: 
0       None
1       None
2       None
3       None
4       None
        ... 
9995    None
9996    None
9997    None
9998    None
9999    None
Length: 10000, dtype: object

In [3]: thread_ids
Out[3]: {140372742666048}

However, nowadays several pandas operations release the GIL or allow for various degrees of parallelism under the hood, see this GitHub comment.

user_5543190
Related