I have a dataframe df whose the sets of column and row names are the same, i.e., markers = ['a', 'b', 'c']. I have a function func that takes a pair of strings and produces a number. Now I want to fill in the dataframe with values produced by func. My approach is by using loop, i.e.,
for m in markers:
for n in markers:
df.loc[m, n] = func(m, n)
This is clearly not efficient. Could you elaborate how to do so with dummy.Pool? Previously, I only use dummy.Pool to get a list of outputs from a list of inputs.
import os
import pandas as pd
from multiprocessing import dummy
core = os.cpu_count() # The number of logical cores of my CPU
P = dummy.Pool(processes = core)
# Function that takes a pair of strings and produces a number
def func(m, n):
if m == n:
return(1)
else:
return(0)
markers = ['a', 'b', 'c'] # The sets of row and column names are the same
df = pd.DataFrame(data = 1, index = markers, columns = markers)
# Fill in each cell of the dataframe
for m in markers:
for n in markers:
df.loc[m, n] = func(m, n)
df
In my case, I got an warning about performance:
PerformanceWarning: DataFrame is highly fragmented. This is usually the result of calling `frame.insert` many times, which has poor performance. Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()`