How do I replace multiple words within each row of a column that contains full sentences?

user_8933347 0
Viewed 89

I have a data frame (let's call it 'littletext') that has a column with sentences within each row. I also have another table (let's call it 'littledict') that I would like to use as a reference by which to find and replace words and/or phrases within each row of 'littletext'.

Here are my two data frames. I am hard-coding them in this example but will load them as csv files in "real life":

raw_text = {
    "text": ["Hello, world!", "Hello, how are you?", "This world is funny!"],
    "col2": [0,1,1]}

littletext = pd.DataFrame(raw_text, index = pd.Index(['A', 'B', 'C'], name='letter'), columns = pd.Index(['text', 'col2'], name='attributes'))


raw_dict = {
    "key": ["Hello", "This", "funny"],
    "replacewith": ["Hi", "That", "hilarious"]}

littledict = pd.DataFrame(raw_dict, index = pd.Index(['a','b','c'], name='letter'), columns = pd.Index(['key', 'replacewith'], name='attributes'))


print(littletext) # ignore 'col2' since it is irrelevant in this discussion

                   text  col2
A         Hello, world!     0
B   Hello, how are you?     1
C  This world is funny!     1

print(littledict)
     key replacewith
a  Hello          Hi
b   This        That
c  funny   hilarious

I would like to have 'littletext' modified as per below wherein Python will look at more than one word within each sentence of my 'littletext' table (dataframe) and replace multiple words, acting on all rows. The final product should be that 'Hello' has been replaced by 'Hi' in lines A and B, and 'That' was replaced with 'This' and 'funny' was replaced with 'hilarious', both within line C:

                   text      col2
A         Hi, world!            0
B   Hi, how are you?            1
C  That world is hilarious!     1

Here are two attempts that I have tried but neither of which work. They are not generating errors, they are just not modifying 'littletext' as I described above. Attempt #1 'technically' works but it is inefficient and therefore useless for large-scale jobs because I would have to anticipate and program every possible sentence I would need to replace other sentence. Attempt #2 simply does not change anything at all.

My two Attempts that do NOT work are:

Attempt #1: this is not helpful because to use it, I would have to program entire sentences to replace other sentences, which is pointless:

littltext['text'].replace({'Hello, world!': 'Hi there, world.', 'This world is funny!': 'That world is hilarious'})

Attempt #1 returns:

Out[125]: 
0           Hi there, world.
1        Hello, how are you?
2    That world is hilarious
Name: text, dtype: object

Attempt #2: this attempt is closer to the mark but returns no changes whatsoever:

for key in littledict:
    littletext = littletext.replace(key,littledict[key])

Attempt #2 returns:

                   text  col2
0         Hello, world!     0
1   Hello, how are you?     1
2  This world is funny!     1

I have scoured the internet, including Youtube, Udemy, etc., but to no avail. Numerous 'tutorial' sites only cover individual text examples, not entire columns of sentences like the example I am showing and are therefore useless in scaling up to industry-size projects. I am hoping someone can graciously shed light on this since this kind of text manipulation is commonplace in many industry settings.

My humble thanks and appreciation to anyone who can help!!

pandas text python replace string
2 Answers

you were pretty close with the first attempt. You can create the dictionary from littledict with key in index and use regex=True.

print (littletext['text']
          .replace(littledict.set_index('key')
                             ['replacewith'].to_dict(), 
                   regex=True)
      )
0                  Hi, world!
1            Hi, how are you?
2    That world is hilarious!
Name: text, dtype: object
user_92747320

dict littledict to enable you generate a regex and use the regex in .replace.str()to replace the characters you need as follows

s=dict(zip(littledict.key,littledict.replacewith))
littletext['text'].str.replace('|'.join(s), lambda x: s[x.group()])

0                  Hi, world!
1            Hi, how are you?
2    That world is hilarious!
Name: text, dtype: object
user_89869750
Related