Add in a new column that appends all uppercase words in a phrase to a list per row

user_7788837 0
Viewed 63

I have a dataset similar to the following:

      Date                          Sentence                                      Text              Verified
_               
0   2020-01-18 00:00:00 LUKE Diamond is a famous                         Updates · BREAKING News    False
1   2020-01-18 00:00:00 Blog - TASTY YUMMIES                             Brush with ...             False
2   2020-01-18 00:00:00 ACNE Alternative Remedies: Manuka HONEY          Learn more from WEBMD      False
3   2020-01-18 00:00:00 Looking back at 10 YEARS                         As the LOCAL community     False

I would like select words in upper case that might be in Sentence or in Text (I would like to save these results into two separate columns, rows by rows).

      Date                          Sentence                                      Text              Verified    CS                            CT
_               
0   2020-01-18 00:00:00 LUKE Diamond is a famous                         Updates · BREAKING News    False     ['LUKE']               ['BREAKING']
1   2020-01-18 00:00:00 Blog - TASTY YUMMIES                             Brush with ...             False    ['TASTY', 'YUMMIES']            []
2   2020-01-18 00:00:00 ACNE Alternative Remedies: Manuka HONEY          Learn more from WEBMD      False      ['ACNE','HONEY']      ['WEBMD']
3   2020-01-18 00:00:00 Looking back at 10 YEARS                         As the LOCAL community     False      ['YEARS']             ['LOCAL']

I have tried as follows (this should generate a list of words in upper cases, rows by rows), but I am getting the following error: TypeError: expected string or bytes-like object in caps. Could you please tell me what is not working and how I can get the expected output? Many thanks.

import re

def s_l(file):
    s = []

    for c in file['Sentence'].tolist(): 
        caps = re.findall('([A-Z]+(?:(?!\s?[A-Z][a-z])\s?[A-Z])+)', c)
        s.append(caps)
    for c in file['Text'].tolist(): 
        caps = re.findall('([A-Z]+(?:(?!\s?[A-Z][a-z])\s?[A-Z])+)', c)
        s.append(caps)

    file['CS'] = pd.Series(s)
    s = [x for x in s if x != [] and len(x)>1]
    file['CT'] = pd.Series(s)
    s = [x for x in s if x != [] and len(x)>1]

    return file, s

s_df, s =s_l(df)

I think I am all wrongly considering a list of words, so probably I should add another list (not only s).

pandas python regex
2 Answers

Create a list comprehension object m that compares values to .upper() to get all uppercase letters and .isalpha() to make sure you are not bringing in strings / numbers where .upper() doesn't do anything to them. Then, simply create new columns that utilize the list comprehension with .apply(m)

m = lambda x: [y for y in str(x).split(' ') if y.upper() == y and y.isalpha()]
df['CS'] = df['Sentence'].apply(m)
df['CT'] = df['Text'].apply(m)

output:

    Date                          Sentence                                      Text              Verified    CS                            CT               
0   2020-01-18 00:00:00 LUKE Diamond is a famous                         Updates · BREAKING News    False     ['LUKE']               ['BREAKING']
1   2020-01-18 00:00:00 Blog - TASTY YUMMIES                             Brush with ...             False    ['TASTY', 'YUMMIES']            []
2   2020-01-18 00:00:00 ACNE Alternative Remedies: Manuka HONEY          Learn more from WEBMD      False      ['ACNE','HONEY']      ['WEBMD']
3   2020-01-18 00:00:00 Looking back at 10 YEARS                         As the LOCAL community     False      ['YEARS']             ['LOCAL']
user_63667700

You can regex with findall

df['CS'] = df.Sentence.str.findall('\\b[A-Z]+\\b')

Output

df['CS']

0              [LUKE]
1    [TASTY, YUMMIES]
2       [ACNE, HONEY]
3             [YEARS]
user_79645270
Related