'PerformanceWarning: DataFrame is highly fragmented' whilst creating a dataframe of with 10K columns

Viewed 334

Have viewed similar solutions like the ones here but I can't get it to work on my implementation.

referee referee_phone_number referred_by
referee_A 01500600 Person A_fraudster
referee_B 01500601 Person A_fraudster
referee_C 01500602 Person A_fraudster
referee_D 01500603 Person A_fraudster
referee_E 01501928 Person B_non_fraudster
referee_F 01779810 Person B_non_fraudster

Using code below:

from similarity.normalized_levenshtein import NormalizedLevenshtein
normalized_levenshtein = NormalizedLevenshtein()

df = pd.DataFrame({'numbers': df.referee_phone_number})
numbers = df.referee_phone_number.tolist()

for i in numbers:
    df[i] = df.referee_phone_number.apply(lambda x: normalized_levenshtein.similarity(x, i))
df.set_index('numbers', inplace=True)
df = df.where((df[numbers] > 0.5) & (df[numbers] < 1), other=0)
df = df.max(axis=1)
df = df.reset_index()
df.rename(columns={ 0: 'max_similarity_within_dataset'}, inplace=True)
df

However, I am coming into problems at this point in the code where the loop is:

for i in numbers:
    df[i] = df.referee_phone_number.apply(lambda x: normalized_levenshtein.similarity(x, i))

Which is supposed to first generate this matrix:

numbers 019500600 019500601 019500602 019500603 014501928 016779810
019500600 1 0.888889 0.888889 0.888889 0.444444 0.333333
019500601 0.888889 1 0.888889 0.888889 0.444444 0.222222
019500602 0.888889 0.888889 1 0.888889 0.444444 0.222222
019500603 0.888889 0.888889 0.888889 1 0.444444 0.222222
014501928 0.444444 0.444444 0.444444 0.444444 1 0.222222
016779810 0.333333 0.222222 0.222222 0.222222 0.222222 1

My code implementation works in small number of rows, but in the case of doing it for 11K phone numbers it means creating a matrix that is 11K*11K. I hit a performanceWarning:

<ipython-input-56-f4893743bb4a>:11: PerformanceWarning: DataFrame is highly fragmented.  This is usually the result of calling `frame.insert` many times, which has poor performance.  Consider using pd.concat instead.  To get a de-fragmented frame, use `newframe = frame.copy()`
  df[i] = df.referee_phone_number.apply(lambda x: normalized_levenshtein.similarity(x, i))

If there any way I can write my code more efficiently?

0 Answers
Related