I am using google colab to perform fuzzy match between 2 columns of a dataframe
I want to list all values in first column based on complete or partial match and put EXISTS if there's a match.
I have tried below, but the code take very long to execute on 5000 * 2
records
Below is my code :
#pip install fuzzywuzzy
import pandas as pd
from fuzzywuzzy import fuzz
from fuzzywuzzy import process
import difflib
data=pd.read_csv('/content/mydata')
Df=pd.DataFrame(data[['ColA','ColB']])
df1=pd.DataFrame(data['ColA'])
df2=pd.DataFrame(data['ColB'])
def fuzzy_merge(df_1, df_2, key1, key2, threshold=90, limit=2):
"""
:param df_1: the left table to join
:param df_2: the right table to join
:param key1: key column of the left table
:param key2: key column of the right table
:param threshold: how close the matches should be to return a match, based on Levenshtein distance
:param limit: the amount of matches that will get returned, these are sorted high to low
:return: dataframe with boths keys and matches
"""
s = df_2[key2].tolist()
m = df_1[key1].apply(lambda x: process.extract(x, s, limit=limit))
df_1['matches'] = m
m2 = df_1['matches'].apply(lambda x: ', '.join([i[0] for i in x if i[1] >= threshold]))
df_1['matches'] = m2
return df_1
fuzzy_merge(df1,df2,'ColA','ColB')
Below is my Dataframe
|ColA| ColB| result|
|-|-|-|
|aaabc.eval.moc| abcde| EXISTS|
|abcde.eval| abc.123| EXISTS|
|def.gcd.xyz| def.gc| EXISTS|
|abc.123.moc| xyz123.eval.moc.facebook.google| EXISTS|
|xyz123.eval.moc| google.facebook.apple.chromebook| EXISTS|
|google.facebook.apple| 435 | NOT EXISTS|
|Testing435| `|NOT EXISTS`