Have viewed similar solutions like the ones here but I can't get it to work on my implementation.
| referee | referee_phone_number | referred_by |
|---|---|---|
| referee_A | 01500600 | Person A_fraudster |
| referee_B | 01500601 | Person A_fraudster |
| referee_C | 01500602 | Person A_fraudster |
| referee_D | 01500603 | Person A_fraudster |
| referee_E | 01501928 | Person B_non_fraudster |
| referee_F | 01779810 | Person B_non_fraudster |
Using code below:
from similarity.normalized_levenshtein import NormalizedLevenshtein
normalized_levenshtein = NormalizedLevenshtein()
df = pd.DataFrame({'numbers': df.referee_phone_number})
numbers = df.referee_phone_number.tolist()
for i in numbers:
df[i] = df.referee_phone_number.apply(lambda x: normalized_levenshtein.similarity(x, i))
df.set_index('numbers', inplace=True)
df = df.where((df[numbers] > 0.5) & (df[numbers] < 1), other=0)
df = df.max(axis=1)
df = df.reset_index()
df.rename(columns={ 0: 'max_similarity_within_dataset'}, inplace=True)
df
However, I am coming into problems at this point in the code where the loop is:
for i in numbers:
df[i] = df.referee_phone_number.apply(lambda x: normalized_levenshtein.similarity(x, i))
Which is supposed to first generate this matrix:
| numbers | 019500600 | 019500601 | 019500602 | 019500603 | 014501928 | 016779810 |
|---|---|---|---|---|---|---|
| 019500600 | 1 | 0.888889 | 0.888889 | 0.888889 | 0.444444 | 0.333333 |
| 019500601 | 0.888889 | 1 | 0.888889 | 0.888889 | 0.444444 | 0.222222 |
| 019500602 | 0.888889 | 0.888889 | 1 | 0.888889 | 0.444444 | 0.222222 |
| 019500603 | 0.888889 | 0.888889 | 0.888889 | 1 | 0.444444 | 0.222222 |
| 014501928 | 0.444444 | 0.444444 | 0.444444 | 0.444444 | 1 | 0.222222 |
| 016779810 | 0.333333 | 0.222222 | 0.222222 | 0.222222 | 0.222222 | 1 |
My code implementation works in small number of rows, but in the case of doing it for 11K phone numbers it means creating a matrix that is 11K*11K. I hit a performanceWarning:
<ipython-input-56-f4893743bb4a>:11: PerformanceWarning: DataFrame is highly fragmented. This is usually the result of calling `frame.insert` many times, which has poor performance. Consider using pd.concat instead. To get a de-fragmented frame, use `newframe = frame.copy()`
df[i] = df.referee_phone_number.apply(lambda x: normalized_levenshtein.similarity(x, i))
If there any way I can write my code more efficiently?