Alternatives to .apply() to increase performance speed

user_14538164 0
Viewed 150

I wrote a function that cleans the text of a given string. And I apply this function to a column in a dataframe(~ 1 million records).

from cleanco import cleanco
import re
import pandas as pd
import unidecode

def clean_name(text):

     #convert plain text to utf-8
     text = unidecode.unidecode(text)

     text = text.upper()
     text = re.sub('[()]', '', text)
     text = text.replace(',' , '')
     text = text.replace(' - ', ' ')
     text = text.replace(r"\(.*\)","")
     text = text.replace(' AND ', ' & ')
     text = text.strip()
 
     x = cleanco(text)
     text = x.clean_name()
     text = text.replace('.', '')
     text = text.title()
    
     return text

df['name_clean'] = customer_info_df['name'].apply(utils.clean_name)

While the function gets the job done, the performance speed seems to be a bit long (~13-15mins) to finish a dataframe of over 1 million rows. Are there any alternatives to increase the speed and effeciency?

pandas python regex
1 Answers

This isn't the best solution, but one approach to reduce the time processing is the concatenation of the replace method.

import re

text = text.upper()
text = re.sub('[()]', '', text)
text = text.replace(',' , '').replace(' - ', ' ').replace(r"\(.*\)","").replace(' AND ', ' & ').strip()

In honor of true, this method takes two subcases performances.

Case 1: strings are no-numeric

With concatenation 

text ='a.b-c.d-d.d.dadad.da-da.dakdapsckcpkepéka´pksXCAKXosñ{xas,x_x.-cda,xa-.-lksslakxslñksaxñlax,-.'
start = datetime.now()
text = text.upper()
text = re.sub('[()]', '', text)
text = text.replace(',' , '').replace(' - ', ' ').replace(r"\(.*\)","").replace(' AND ', ' & ').strip()
end = datetime.now()

start - end = 0.000101

Original 

start = datetime.now()
text = 'a.b-c.d-d.d.dadad.da-da.dakdapsckcpkepéka´pksXCAKXosñ{xas,x_x.-cda,xa-.-lksslakxslñksaxñlax,-.'
text = text.upper()
text = re.sub('[()]', '', text)
text = text.replace(',' , '')
text = text.replace(' - ', ' ')
text = text.replace(r"\(.*\)","")
text = text.replace(' AND ', ' & ')
text = text.strip()
end = datetime.now()

start - end = 0.000202

user_109862090
Related