How to search an element in column of a dataframe containing lists

user_12009028 0
Viewed 278

I have a pandas dataframe df which contains two columns. First column sentence contains the sentence and second column keywords contains the all the keywords in list from the sentence in the first column. So my dataframe looks something like this :

>>> df
                                       sentence                          keywords
0        What are the advantages of the prepaid             [advantages, prepaid]
1     is salon facility available in your hotel          [salon, facility, hotel]
2  I want to check my balance and usage history  [check, balance, usage, history]

Now suppose I have a list like ["salon", "usage", "history"] containing the keywords which I want to search against the keywords column and in return I want to get all the sentences containing the keywords in the given list. So the output in this particular case would be :

>>> output
["is salon facility available in your hotel", "I want to check my balance and usage history"]

What is the fastest and most efficient way of achieving this result because I have thousands of sentences in the sentence column. The only way I know to achieve this to get all the lists from keywords column in a separate list and enumerate over it and get the indexes of the all the sub-lists containing the keywords and then look for same indexes in the sentence column. But this would be very slow. Is there any other better way of doing this?

Data:

data = {'sentence': ['What are the advantages of the prepaid',
                     'is salon facility available in your hotel',
                     'I want to check my balance and usage history'],
        'keywords': [['advantages', 'prepaid'],
                     ['salon', 'facility', 'hotel'],
                     ['check', 'balance', 'usage', 'history']]}
pandas dataframe python
2 Answers

You can use set.isjdisoint and check if there are any overlapping words with the "keywords" to create a boolean mask. Then, use the mask on df to filter the matching sentences:

lst = ["salon", "usage", "history"]
msk = df['keywords'].apply(lambda x: not set(x).isdisjoint(lst))
out = df.loc[msk, 'sentence'].tolist()

Output:

['is salon facility available in your hotel', 'I want to check my balance and usage history']
Admin1

For each row you should check the set intersection (or difference) to determine which sentences' keywords overlap with the search keywords:

search_words = {'salon', 'usage', 'history'}

def search(keywords, search_words):
  return not search_words.isdisjoint(keywords)

results = df['keywords'].apply(search, args=(search_words,))

df.loc[results, 'sentence'].tolist()
user_21536360
Related