"PerformanceWarning: DataFrame is highly fragmented. This is usually the result of calling `frame.insert` many times, which has poor performance."

user_12543122 0
Viewed 4862

Essentially I'm ranking each column in the df dataframe and adding it to the ranking dataframe. I'm obviously not doing this very efficiently, and I'm wondering if someone can point me in the correct direction.

for x in range(1,num_sims+1):
    ranking[x] = df[x].rank(ascending=False, method='min')

The full error message is:

PerformanceWarning: DataFrame is highly fragmented.  This is usually
the result of calling `frame.insert` many times, which has poor
performance.  Consider using pd.concat instead.  To get a
de-fragmented frame, use `newframe = frame.copy()`   ranking[x] =
df[x].rank(ascending=False, method='min')"
pandas dataframe concatenation python
1 Answers

Sample that reproduces the warning:

import numpy as np
import pandas as pd

# Sample `df`
np.random.seed(5)
df = pd.DataFrame(np.random.randint(1, 100, (4, 5000)))
df.columns = df.columns + 1

num_sims = len(df.columns)  # Placeholder for `num_sims`
ranking = pd.DataFrame()  # Placeholder for `ranking`

for x in range(1, num_sims + 1):
    ranking[x] = df[x].rank(ascending=False, method='min')

PerformanceWarning: DataFrame is highly fragmented. This is usually the result of calling frame.insert many times, which has poor performance. Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use newframe = frame.copy() ranking[x] = df[x].rank(ascending=False, method='min')

Fix using DataFrame.rank and concat instead:

ranking = pd.DataFrame()  # Placeholder for `ranking`
ranking = pd.concat(
    [ranking, df[range(1, num_sims + 1)].rank(ascending=False, method='min')],
    axis=1
)

output without error:

   1     2     3     4     5     6     ...  4995  4996  4997  4998  4999  5000
0   2.0   2.0   3.0   1.0   4.0   1.0  ...   3.0   3.0   3.0   4.0   4.0   1.0
1   1.0   1.0   4.0   2.0   3.0   2.0  ...   4.0   1.0   2.0   3.0   1.0   3.0
2   3.0   4.0   1.0   4.0   1.0   4.0  ...   1.0   4.0   1.0   1.0   3.0   4.0
3   4.0   3.0   2.0   3.0   2.0   3.0  ...   2.0   2.0   3.0   2.0   2.0   2.0

*Naturally, if ranking is empty we can just create it directly from df:

ranking = df[range(1, num_sims + 1)].rank(ascending=False, method='min')

Sanity check that they produce the same results:

import numpy as np
import pandas as pd

np.random.seed(5)
df = pd.DataFrame(np.random.randint(1, 100, (4, 5000)))
df.columns = df.columns + 1
ranking = pd.DataFrame()
num_sims = len(df.columns)

for x in range(1, num_sims + 1):
    ranking[x] = df[x].rank(ascending=False, method='min')

print(ranking.eq(pd.concat(
    [pd.DataFrame(),
     df[range(1, num_sims + 1)].rank(ascending=False, method='min')],
    axis=1
)).all(axis=None))  # True
user_154978880
Related