Efficiently apply function on a large pandas dataframe

user_7547047 0
Viewed 1999

I am trying to create a new column by applying a function (jaccard string distance) between two columns of a pandas dataframe. My dataframe has around 400 million rows. 

df['sim_scr'] = df.apply(lambda x: jaccard(x[0],x[1],2),axis=1)

Owing to the huge size, it is taking quite some time, so i tried splitting the dataframe to chunks, apply the function and join them back. Even this is taking very large time:

for chunk in np.array_split(df,1000):
 chunk['sim_scr'] = chunk.apply(lambda x: jaccard(x[0],x[1],2),axis=1)
 df2 = df2.append(chunk)

Is there any efficient way to achieve this?

EDIT:

This is how i defined my Jaccard distance function:

def jaccard(a,b,n):
 s = [a[i:i+n] for i in range(len(a)-n+1)]
 t = [b[i:i+n] for i in range(len(b)-n+1)]
 if len(list(set(s) | set(t))) >0:
  jac_coeff = 1 -  len(list(set(s) & set(t))) / len(list(set(s) | set(t)))
 else:
 jac_coeff = 1
return jac_coeff

Sample Dataframe:

nm1          nm2
John A       Smith K
California   Cadifornia
San Frans    San Fransisco
pandas apply
1 Answers

In my understanding, operation with vector is the most effective way. If you split your large rows of vector into severals and calculate, I think it will be slower in total.

And, if you split and merge, it will take some time to copy/merge the large vector.

How about this way, convert you 2 dim vector n*2 into 3 dim, like n/5 * 5 * 2, and find another function or lib which can work one that vector.

user_9666980
Related