Improve performance of a nested apply in pandas

user_4034593 0
Viewed 319

I have a pandas DataFrame names and a Series illegal_words.

# names - ca. 250k rows
                    name
0   MISS ELFRIEDA ALPERT
1         DALE VON PETTY
2  MOHAMMAD IBN MASILLAH
3      YELENA THE MORRIS
4      MR. SHENNA DEMOSS
...

# illegal_words - ca. 2k rows
0  MISS
1  VON
2  THE
...

I want to remove any illegal word from names.

I'm calling a loop in a loop:

import re

for word in illegal_words:
    names['name'] = names['name'].apply(lambda x: re.sub(word, '', x))

Output:

# names
                 name
0     ELFRIEDA ALPERT
1         DALE  PETTY
2  MOHAMMAD  MASILLAH
3      YELENA  MORRIS
4       SHENNA DEMOSS
...

(double spaces are not much of an issue)

And it works, but... this is very slow. The re.sub() method is called 2'000 * 250'000 = 500'000'000 times!

What can I do to speed it up?

(since the illegal_words is very project-specific, I cannot use any external package)

pandas python performance
1 Answers

Can you try that?

illegal_words = ['MISS', 'VON', 'THE']

out = df['name'].str.replace(fr"({'|'.join(illegal_words)}) ", '', regex=True)

>>> out
0          ELFRIEDA ALPERT
1               DALE PETTY
2    MOHAMMAD IBN MASILLAH
3            YELENA MORRIS
4        MR. SHENNA DEMOSS
Name: name, dtype: object

Performance

For a random list of 2,500 words and 250,000 records:

%timeit df['name'].str.replace(fr"({'|'.join(illegal_words)}) ", '', regex=True)
130 ms ± 870 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
user_152399510
Related