Multiprocessing: fill a dataframe with values produced by a function that takes row and column names as input?

Viewed 45

I have a dataframe df whose the sets of column and row names are the same, i.e., markers = ['a', 'b', 'c']. I have a function func that takes a pair of strings and produces a number. Now I want to fill in the dataframe with values produced by func. My approach is by using loop, i.e.,

for m in markers:
    for n in markers:
        df.loc[m, n] = func(m, n)

This is clearly not efficient. Could you elaborate how to do so with dummy.Pool? Previously, I only use dummy.Pool to get a list of outputs from a list of inputs.

import os
import pandas as pd
from multiprocessing import dummy

core = os.cpu_count() # The number of logical cores of my CPU
P = dummy.Pool(processes = core)

# Function that takes a pair of strings and produces a number
def func(m, n):
    if m == n:
        return(1)
    else:
        return(0)

markers = ['a', 'b', 'c'] # The sets of row and column names are the same
df = pd.DataFrame(data = 1, index = markers, columns = markers)

# Fill in each cell of the dataframe    
for m in markers:
    for n in markers:
        df.loc[m, n] = func(m, n)
        
df    

In my case, I got an warning about performance:

PerformanceWarning: DataFrame is highly fragmented.  This is usually the result of calling `frame.insert` many times, which has poor performance.  Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()`
0 Answers
Related