Compare two columns on Fuzzy match

Viewed 33

I am using google colab to perform fuzzy match between 2 columns of a dataframe

I want to list all values in first column based on complete or partial match and put EXISTS if there's a match.

I have tried below, but the code take very long to execute on 5000 * 2 records

Below is my code :

#pip install fuzzywuzzy
import pandas as pd
from fuzzywuzzy import fuzz
from fuzzywuzzy import process
import difflib
data=pd.read_csv('/content/mydata')
Df=pd.DataFrame(data[['ColA','ColB']])


 
df1=pd.DataFrame(data['ColA'])
df2=pd.DataFrame(data['ColB'])

def fuzzy_merge(df_1, df_2, key1, key2, threshold=90, limit=2):
    """
    :param df_1: the left table to join
    :param df_2: the right table to join
    :param key1: key column of the left table
    :param key2: key column of the right table
    :param threshold: how close the matches should be to return a match, based on Levenshtein distance
    :param limit: the amount of matches that will get returned, these are sorted high to low
    :return: dataframe with boths keys and matches
    """
    s = df_2[key2].tolist()
    
    m = df_1[key1].apply(lambda x: process.extract(x, s, limit=limit))    
    df_1['matches'] = m
    
    m2 = df_1['matches'].apply(lambda x: ', '.join([i[0] for i in x if i[1] >= threshold]))
    df_1['matches'] = m2
    
    return df_1

fuzzy_merge(df1,df2,'ColA','ColB')

Below is my Dataframe

|ColA|  ColB|   result|
|-|-|-|
|aaabc.eval.moc|    abcde|  EXISTS|
|abcde.eval|    abc.123|    EXISTS|
|def.gcd.xyz|   def.gc| EXISTS|
|abc.123.moc|   xyz123.eval.moc.facebook.google|    EXISTS|
|xyz123.eval.moc|   google.facebook.apple.chromebook|   EXISTS|
|google.facebook.apple| 435 |   NOT EXISTS|
|Testing435|   `|NOT EXISTS`
0 Answers
Related