Get jaccard distance between rows without preformong one hot encoding on the column in python

user_7269177 0
Viewed 311

I have a table that looks like that:

id   feature_1  feature_2  feature_3
1       A          x          r
23      A          v          r
56      B          z          r

I want to create some sort of Jaccard distance between the rows (ids) in an efficent way and without converting the feature to one hot encoding cols (a lot of possibilities).

How can I get a result of sum of the similarity between rows? something like that:

id_1  id_2  similarity  num_of_features  distance
 1     23       2              3            2/3
 1     56       1              3            1/3
 23    56       1              3            1/3

My code:

def create_pairs(ids_list):
  pairs_list = []
  for (p1,p2) in itertools.combinations(ids_list,2):
      pair = [p1,p2]
      pairs_list.append(pair)

  return pairs_list

def get_distance(id_1, id_2, df):

    ????

    return distance


ids_list= list(df['id'].unique())
pairs_list = create_pairs(ids_list=ids_list)
pairs_df = pd.DataFrame(pairs_list,columns=['id_1','id_2'])
distance_list = []
for [id_1, id_2] in pairs_list:
  distance = get_distance(id_1=id_1, id_2=id_2,df=df)

pairs_df['distance'] = distance_list
distance python
2 Answers

I believe there is no row-wise built in functionality for jaccard distance in scikit, but you can try out following:

from sklearn.metrics import jaccard_score
import pandas as pd 

data = [['a', 'x', 'r'], ['a', 'v', 'r'], ['b', 'z', 'r']] 
  
# Create the pandas DataFrame 
df = pd.DataFrame(data, columns = ['feature_1',  'feature_2',  'feature_3'])

print(df)
df_cmp = df.copy()
for index, row in df.iterrows():
  for index_, row_ in df_cmp.iterrows():
    if index == index_:
      continue
    print(index, index_, jaccard_score(row,row_, average='weighted'))
  df_cmp = df_cmp.drop(index) #remove after as done already

Create a copy you will work with - so you remove already passed rows. Then you can calculate jaccard distances for rows. Output:

  feature_1 feature_2 feature_3
0         a         x         r
1         a         v         r
2         b         z         r

And scores for lines:

0 1 0.6666666666666666
0 2 0.3333333333333333
1 2 0.3333333333333333

You can instead of printing write it to a dataframe and work with it later. This method however has complexity O(n*log(n)), maybe there will be something faster.

EDIT:

I somehow didn't notice you have pairs already created, thus just replace your question marks by:

distance = jaccard_score(df.iloc[id_1],df.iloc[id_2], average='weighted')
user_31574280

Try this, it should be faster:

def get_distance(id_1, id_2, df):

    similarity_count = df.loc[id_1]==df.loc[id_2]
    similarity_count = similarity_count.sum(0)
    distance = 1-(similarity_count /len(list(df.columns)))

    return distance
user_81153450
Related