extend() not producing a list

user_14100757 0
Viewed 102

I am working with a list of strings and a dataframe containing strings. Imagine the scenario:

A = ['the', 'a', 'with', 'from', 'on']
and a dataframe: 
df = {'col1':['string', 'string'], 'col2':['the man from a town', 'the man on a bus']}

I am trying to now create a new column in my data_frame that would present values in column 2 of my data_frame which are in my list A (in this case: the, from, a)

What I wrote was this:

def words_in_A(row): 
     res=[]
     for item in A:
          if item in row:
              res.extend(item) 
              return res

df[col3] = df[col2].apply(lambda x: words_in_A(x))

I would expect the output to be a list with multiple values: 

col 1          col2                   col3
string         the man from a town    'the', 'from', 'a'
string         the man on a bus       'the', 'on', 'a'

But the function just returns the last item ('a') rather than a list. I am not sure why this use of extend() isn't producing a list for me. Please help!

pandas python
3 Answers

Your code just needed a little indentation adjustment and use append instead of extend. If you extend then the string 'the' will be taken as a list and each letter will be appended to the collecting list.

def words_in_A(row): 
    lst = []
    for item in A:
        if item in row:
            lst.append(item) 
    return lst

I honestly though that a list comprehension and even Shubham's answer with regex would be faster than apply, but I stand corrected. Here's the timings with your dataframe but with 20,000 rows instead of 2.

with apply 0.078s
with list comp 0.076s
with regex 0.168s
with regex, no join 0.141s

And the testing code

from time import time

t0 = time()
df['col3'] = df['col2'].apply(words_in_A)
print('with apply', f'{time() - t0:.3f}s')

t0 = time()
df['col3'] = [[item for item in A if item in row] for row in df.col2]
print('with list comp', f'{time() - t0:.3f}s')

t0 = time()
pat = rf"(?i)\b(?:{'|'.join(A)})\b"
df['col3'] = df['col2'].str.findall(pat).str.join(', ')
print('with regex', f'{time() - t0:.3f}s')

t0 = time()
pat = rf"(?i)\b(?:{'|'.join(A)})\b"
df['col3'] = df['col2'].str.findall(pat)
print('with regex, no join', f'{time() - t0:.3f}s')

Output

         col1                 col2          col3
0      string  the man from a town  the, from, a
1      string    a person on a bus      a, on, a
2      string  the man from a town  the, from, a
3      string    a person on a bus      a, on, a
4      string  the man from a town  the, from, a
...       ...                  ...           ...
19995  string    a person on a bus      a, on, a
19996  string  the man from a town  the, from, a
19997  string    a person on a bus      a, on, a
19998  string  the man from a town  the, from, a
19999  string    a person on a bus      a, on, a

[20000 rows x 3 columns]
user_66928980

extend(): Iterates over its argument and adding each element to the list and extending the list.

So x.extend("one") will result in ['o','n','e'] what you instead need is x.append which appends one to the end of list x.

Also you are populating list called res so you have to return it after you are done doing operations on it.

A = ['the', 'a', 'with', 'from', 'on']
df = {'col1':['string', 'string'], 'col2':['the man from a town', 'a person on a bus']}
df = pd.DataFrame(df)

def words_in_A(row): 
  res=[]
  for item in A:
    if item in row:
      res.append(item) 
  return res

df['col3'] = df['col2'].apply(lambda x: words_in_A(x))
print (df)

Output:

     col1                 col2            col3
0  string  the man from a town  [the, a, from]
1  string    a person on a bus         [a, on]

Pythonic:

df['col3'] = df['col2'].apply(lambda x: list(set(x.split()).intersection(A)))
user_4239260

Use Series.str.findall with a regex pattern to find all the matching values from list A then use Series.str.join:

pat = rf"(?i)\b(?:{'|'.join(A)})\b"
df['col3'] = df['col2'].str.findall(pat).str.join(', ')

Result:

     col1                 col2          col3
0  string  the man from a town  the, from, a
1  string     the man on a bus    the, on, a
user_128331660
Related