Some context: I'm currently working on data on Fraud Detection and my data looks like the following with relevant columns:
| referee | referee_phone_number | referred_by |
|---|---|---|
| referee_A | 019500600 | Person A_fraudster |
| referee_B | 019500601 | Person A_fraudster |
| referee_C | 019500602 | Person A_fraudster |
| referee_D | 019500603 | Person A_fraudster |
| referee_E | 014501928 | Person B_non_fraudster |
| referee_F | 016779810 | Person B_non_fraudster |
Notice that the phone numbers referred by Person A are similar. Fraudsters tend to register phone numbers en masse.
One feature that I would like to engineer would be string similarity grouped by referred_by, which looks something like this, with approximates for the average value similarities to be fed into my decision tree, with the goal of using the value for training, since non-frausters will low phone number similarities:
| referred_by | average_phone_number_similarity |
|---|---|
| Person A | 0.90 |
| Person B | 0.20 |
I've tried exploring Levenstein Distance and have the following script using difflib , thinking that I could pit two lists together:
string_list_1 = ['02223785428', '02223785390', '02223784947', '0165104490']
string_list_2 = ['02223785428', '02223785390', '02223784947']
from difflib import SequenceMatcher
similar = []
not_similar = []
for item1 in string_list_1:
# Set the state as false
found = False
for item2 in string_list_2:
if SequenceMatcher(None, a=item1,b=item2).ratio() > 0.8:
similar.append(item1)
found = True
break
if not found:
not_similar.append(item1)
# Making sure that there is not duplcates in the list
print("Similar : ", list(dict.fromkeys(similar)))
print("Not Similar : ", list(dict.fromkeys(not_similar)))
>> Similar : ['02223785428', '02223785390', '02223784947']
Not Similar : ['0165104490']
However, it doesn't solve my issue, especially when it comes to computing similarity between each phone number, I don't think I am getting it quite right, can anyone lend a hand?