Error:
"DataFrame is highly fragmented. This is usually the result of calling
frame.insertmany times, which has poor performance. Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()"
Code:
df_DIFFs = pd.DataFrame()
for a, b in itertools.permutations(df.columns,2):
df_DIFFs[f'{a}-{b}'] = df[a] - df[b]
df_DIFFs
I can't imagine how to use pd.concat or any other more efficient solution in this situation. Thanks.
You can follow the advice by storing column differences in a list and then concatenating:
# our test df
df = pd._testing.makeDataFrame()
# here we will store columns that are differences
diff_cols = []
# iterate
for a, b in itertools.permutations(df.columns,2):
tmp = (df[a] - df[b]).rename(f'{a}-{b}')
diff_cols.append(tmp)
# concatenate
df_DIFFs = pd.concat(diff_cols, axis=1)
output
df_DIFFs
output:
A-B A-C A-D B-A B-C B-D C-A C-B C-D D-A D-B D-C
---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ----------
igc6RyqBmG 1.02425 1.39726 -0.460256 -1.02425 0.373015 -1.48451 -1.39726 -0.373015 -1.85752 0.460256 1.48451 1.85752
MMIfUNYx8i 1.18588 3.64252 0.0921551 -1.18588 2.45664 -1.09373 -3.64252 -2.45664 -3.55037 -0.0921551 1.09373 3.55037
2MhnKJeZ2K 0.86183 1.00485 -1.12921 -0.86183 0.143022 -1.99104 -1.00485 -0.143022 -2.13406 1.12921 1.99104 2.13406
in9wt14lku -1.40201 -0.0527898 -1.73563 1.40201 1.34922 -0.333613 0.0527898 -1.34922 -1.68284 1.73563 0.333613 1.68284
7S6tkx2WFN 0.779529 1.40291 0.418794 -0.779529 0.623382 -0.360735 -1.40291 -0.623382 -0.984117 -0.418794 0.360735 0.984117
1C2wmIahde -2.18965 0.161868 -1.36776 2.18965 2.35152 0.821893 -0.161868 -2.35152 -1.52963 1.36776 -0.821893 1.52963
oRqd898jAl 0.658137 1.58394 0.712051 -0.658137 0.925807 0.0539145 -1.58394 -0.925807 -0.871893 -0.712051 -0.0539145 0.871893
...