Marking "True" rows that contain a keyword (keyword being the column name) in pandas in one iteration

user_10233425 0
Viewed 58

I want to do a keyword search on pandas dataframe, adding each keyword as a column to the dataset, and marking "True" rows of data that contain the keyword.

A piece of code that does it nicely is this: 

stocks = ['Microsoft','Apple', 'Amazon']
for stock in stocks:
    df[stock] = df.astype(str).sum(axis=1).str.contains(stock)

However, this loops through the entire dataset once for each keyword. I would like to do the same thing in one iteration (i.e. each row is checked for the presence of the keywords only once), because my dataset is large.

The expected result is this:

Expected dataframe

Any help will be appreciated

EDIT: I am getting a Memory Error.

pandas dataframe python
1 Answers

I can't imagine there is something that does what you want without iterating over the three keywards.

Still, I have two suggestions:

  1. Do it just once: It might be a long operation, but then you dump it to a file, and next time you need it, is already there: load the file and you're good to go.

You can use something simple like csv:

my_df.to_csv("my_file.csv")

or if you want to be faster use the hdf5 format

  1. Define your "text" column as string-type outside the loop. For what I see from the picture, you don't need to sum(axis=1), but if you alctually must, do it also outside the loop: create a series and then loop over it.

For example:

text_series = df[stock].astype(str).sum(axis=1)
for stock in stocks:
   df[stock] = text_series.str.contains(stock)
  1. Compute the operation in batches.

For example:

batch_size = df.shape[0]/10 # e.g. divide your df in 10 chunks
for i in range(0,df.shape[0],batch_size): #batch_size is the step
   for stock in stocks:
      df[i:i+batch_size][stock] = text_series[i:i+batch_size].str.contains(stock)

N.B. your database has to be divisible by 10 otherwise choose another divisor

user_116665600
Related