Pandas affects results of rapidfuzz match?

user_9906395 0
Viewed 199

I am hitting a wall with this. Rapidfuzz delivers different results for string score similarity if I run it within a pandas dataframe and if I run it by itself? Why the results for Adress Similarity 2 and for the last line are different?

from rapidfuzz import process, utils, fuzz
import pandas as pd
import numpy as np

address_a = 'high new technology development zones huainan city anhui province china anhui anhui any city'
address_b = 'industrial park of funan city'

test_anui_data = {'Processed Client Name': ['anhui jinhan clothing co ltd'], 'Processed Aruvio Name': ['anhui jinhan clothing co ltd'], 'Processed Client Address': [address_a], 'Processed Aruvio Address': [address_b],  'Name Similarity': [89.2857142857142],  'Address Similarity': [np.nan]}  
  
# Create DataFrame  
test_anui = pd.DataFrame(test_anui_data)  
test_anui

test_anui= test_anui[(test_anui['Address Similarity'].isnull()) & (test_anui['Address Similarity']!='')]
test_anui['Address Similarity 2'] = fuzz.token_sort_ratio(str(test_anui['Processed Client Address']), str(test_anui['Processed Aruvio Address']))
print('the address similarity is different? ', fuzz.token_sort_ratio(address_a, address_b))
pandas python
1 Answers

The error comes from the fact that you call the entire column when applying fuzz. If you do the following thing, which is to apply fuzz to the individual row, you get the same result:

test_anui= test_anui[(test_anui['Address Similarity'].isnull()) & (test_anui['Address Similarity']!='')]
test_anui['Address Similarity 2'] = fuzz.token_sort_ratio(str(test_anui.at[0,'Processed Client Address']), str(test_anui.at[0,'Processed Aruvio Address']))

print('the address similarity is different? ', fuzz.token_sort_ratio(address_a, address_b))

alternatively, using .loc

test_anui= test_anui[(test_anui['Address Similarity'].isnull()) & (test_anui['Address Similarity']!='')]
test_anui['Address Similarity 2'] = fuzz.token_sort_ratio(str(test_anui.loc[0,'Processed Client Address']), str(test_anui.loc[0,'Processed Aruvio Address']))

print('the address similarity is different? ', fuzz.token_sort_ratio(address_a, address_b))

The output in the dataframe is:

    Processed Client Name         Processed Aruvio Name  \
0  anhui jinhan clothing co ltd  anhui jinhan clothing co ltd   

                            Processed Client Address  \
0  high new technology development zones huainan ...   

        Processed Aruvio Address  Name Similarity  Address Similarity  \
0  industrial park of funan city        89.285714                 NaN   

   Address Similarity 2  
0             28.099174

and of fuzz.token_sort_ratio(address_a, address_b) is 28.099173553719012.

In other words, you need to specify which row you are intending on extracting strings from. I suppose your dataframe consists of several rows, which means you'll have to do this for each row:

for i in len(test_anui):
    test_anui['Address Similarity 2'] = fuzz.token_sort_ratio(str(test_anui.loc[i,'Processed Client Address']), 
    str(test_anui.loc[i,'Processed Aruvio Address']))
user_53636860
Related