Apply regex formatting to list of terms to match with pandas column

user_14386187 0
Viewed 235

I have the following DataFrame:

df = pd.DataFrame({'phrase':['websocket internet is loading','foo bar kangeroo bunny','websocket funny internet','scrape the internet with websocket','another one']})

df
    phrase
0   websocket internet is loading
1   foo bar kangeroo bunny
2   websocket funny internet
3   scrape the internet with websocket
4   another one

I am trying to use regex with Pandas' str.contains() to match phrases with multiple words, but not requiring the exact sequence of those words to match. I would like to match using a list of strings: ['websocket internet', 'foo bunny'].

Expected output:

0     True
1     True
2     True
3     True
4    False

I know I can implement regex like so:

df['phrase'].str.contains(r'^(?=.*\bfoo\b)(?=.*\bbunny\b).*$')
df['phrase'].str.contains(r'^(?=.*\bwebsocket\b)(?=.*\binternet\b).*$')

But what if I have a large list of match terms? How would I format all of those strings to the necessary regex, and is there a way for me to implement multiple regex in one str.contains() function?

pandas python regex
1 Answers

Assuming your word boundaries are whitespace boundaries, you can use

import pandas as pd
df = pd.DataFrame({'phrase':['websocket internet is loading','foo bar kangeroo bunny','websocket funny internet','scrape the internet with websocket','another one']})
search = ['websocket internet', 'foo bunny']
search = [f' {x} ' for l in search for x in l.split()]
df['phrase'].apply(lambda x: any(l for l in search if l in f' {x} '))
# => 0     True
#    1     True
#    2     True
#    3     True
#    4    False

Details:

  • search = [f' {x} ' for l in search for x in l.split()] splits out all non-whitespace chunks from the search list and appends an extra space on both side so as to search for an exact word
  • df['phrase'].apply(lambda x: any(l for l in search if l in f' {x} ')) goes through all values in the phrase column and marks as True those that contain at least one item from the search list. An extra space is added to both start and end of each search string.
user_38329700
Related