Creating a column of match probabilities from hmni package python

user_15909328 0
Viewed 272

I have a dataframe that looks like this

import pandas as pd
from pandas import DataFrame    
df = pd.DataFrame({'CEOThisYr': ['Douglas Davis', 'Doug Davis', 'James Taylor', 'Jane Smith'], ['CEOLastYr': 'Doug Davis', 'James Taylor', 'Jane Smith', 'Sarah Jones']})

My goal is to use the hmni package to create a new column with the match probability scores of each name across rows. My attempted code is:

import hmni
matcher = hmni.Matcher(model='latin')
df['MatchPercent'] = matcher.similarity(df['CEOThisYr'], df['CEOLastYr'])

That returns the error: TypeError: Only string comparison is supported in similarity method

I've tried converting both columns to string just to be sure but that still returns the same error. Any ideas where I'm going wrong?

Update*

Thanks to the helpful comment below, I was able to come up with

import hmni
matcher = hmni.Matcher(model='latin')
df['MatchPercent'] = df.apply(lambda x: matcher.similarity(x['CEOThisYr'], x['CEOLastYr']), axis=1)

I went from 9 seconds to process 1 row to processing approx 5000 rows per second.

pandas dataframe python
1 Answers

According to hmni's docs, similarity accepts twos strs as its first and second arguments. You are trying to pass two pandas.Series, i.e., df['CEOThisYr'] and df['CEOLastYr']. You could try using pandas.DataFrame.apply to apply similarity to each row.

>>> import hmni
>>> import pandas as pd
>>>
>>> df = pd.DataFrame({'CEOThisYr': ['Douglas Davis', 'Doug Davis', 'James Taylor', 'Jane Smith'], 'CEOLastYr': ['Doug Davis', 'James Taylor', 'Jane Smith', 'Sarah Jones']})
>>> df
       CEOThisYr     CEOLastYr
0  Douglas Davis    Doug Davis
1     Doug Davis  James Taylor
2   James Taylor    Jane Smith
3     Jane Smith   Sarah Jones
>>>
>>> matcher = hmni.Matcher(model='latin')
>>> df['MatchPercent'] = df.apply(lambda x: matcher.similarity(x['CEOThisYr'], x['CEOLastYr']), axis=1)
>>> df
       CEOThisYr     CEOLastYr  MatchPercent
0  Douglas Davis    Doug Davis      0.922682
1     Doug Davis  James Taylor      0.000000
2   James Taylor    Jane Smith      0.000000
3     Jane Smith   Sarah Jones      0.000000
user_144210350
Related