df.col.replace() vs df.col.str.replace()

user_8553795 0
Viewed 270

I had a requirement where I had to replace ' " ' with ' ' (remove the double quote). So I tried the following:
Approach 1

test.Name = test.Name.replace('"', '')
test_label.name = test_label.name.replace('"', '')

Both the dataframe had same values so if I try to see the difference between values of both the columns I should get null. But to my surprise it was not null. I tried this: 

set(test.Name) - set(test_label.name)

{'Assaf Khalil, Mrs. Mariana (Miriam")"',
 'Cotterill, Mr. Henry Harry""',
 'Coutts, Mrs. William (Winnie Minnie" Treanor)"',
 'Daly, Miss. Margaret Marcella Maggie""',
 'Dean, Miss. Elizabeth Gladys Millvina""',
 'Hocking, Miss. Ellen Nellie""',
 'Johnston, Master. William Arthur Willie""',
 'Johnston, Mrs. Andrew G (Elizabeth Lily" Watson)"',
 'Katavelas, Mr. Vassilios (Catavelas Vassilios")"',
 'Khalil, Mrs. Betros (Zahie Maria" Elias)"',
 'Lindeberg-Lind, Mr. Erik Gustaf (Mr Edward Lingrey")"',
 'McCarthy, Miss. Catherine Katie""',
 'Moubarek, Mrs. George (Omine Amenia" Alexander)"',
 'Nakid, Mrs. Said (Waika Mary" Mowad)"',
 'Nourney, Mr. Alfred (Baron von Drachstedt")"',
 'Riihivouri, Miss. Susanna Juhantytar Sanni""',
 'Riordan, Miss. Johanna Hannah""',
 'Rosenshine, Mr. George (Mr George Thorne")"',
 'Thomas, Mrs. Alexander (Thamine Thelma")"',
 'Wells, Mrs. Arthur Henry (Addie" Dart Trevaskis)"',
 'Wheeler, Mr. Edwin Frederick""',
 'Willer, Mr. Aaron (Abi Weller")"'}

I could still see " in the values which means replace didn't work. So I tried another approach.
Approach 2

test.Name = test.Name.str.replace('"', '', regex=False)
test_label.name = test_label.name.str.replace('"', '', regex=False)

set(test.Name) - set(test_label.name)
set()

The second approach returned what I expected. So my question is why didn't df.col.replace() the values?

pandas dataframe python
1 Answers

By inspection we can determine the type of df.Name and df.Name.str: 

print(type(df.Name)) # <class 'pandas.core.series.Series'>  
print(type(df.Name.str)) # <class 'pandas.core.strings.StringMethods'>

Then, we can find the documentation for Series and StringMethods here and here, respectively. The following are the signatures for their respective replace methods:

Series.str.replace(self, pat, repl, n=-1, case=None, flags=0, regex=True)

Series.replace(self, to_replace=None, value=None, inplace=False, limit=None, regex=False, method='pad')

Note that Series.replace's regex argument default value is False and the one for Series.str.replace is True. So, if you want both functions to have the result you expect, which is to remove the double quote marks you have to set the regex argument to True for the Series.replace method.

Here is an example comparing the results of Series.replace with regex = False and regex = True with that of Series.str.replace:

import pandas as pd

data = { 

        'Name': 
                [
                    'Assaf Khalil, Mrs. Mariana (Miriam")"',
                    'Cotterill, Mr. Henry Harry""',
                    'Coutts, Mrs. William (Winnie Minnie" Treanor)"',
                    'Daly, Miss. Margaret Marcella Maggie""',
                    'Dean, Miss. Elizabeth Gladys Millvina""',
                    'Hocking, Miss. Ellen Nellie""',
                    'Johnston, Master. William Arthur Willie""',
                    'Johnston, Mrs. Andrew G (Elizabeth Lily" Watson)"',
                    'Katavelas, Mr. Vassilios (Catavelas Vassilios")"',
                    'Khalil, Mrs. Betros (Zahie Maria" Elias)"',
                    'Lindeberg-Lind, Mr. Erik Gustaf (Mr Edward Lingrey")"',
                    'McCarthy, Miss. Catherine Katie""',
                    'Moubarek, Mrs. George (Omine Amenia" Alexander)"',
                    'Nakid, Mrs. Said (Waika Mary" Mowad)"',
                    'Nourney, Mr. Alfred (Baron von Drachstedt")"',
                    'Riihivouri, Miss. Susanna Juhantytar Sanni""',
                    'Riordan, Miss. Johanna Hannah""',
                    'Rosenshine, Mr. George (Mr George Thorne")"',
                    'Thomas, Mrs. Alexander (Thamine Thelma")"',
                    'Wells, Mrs. Arthur Henry (Addie" Dart Trevaskis)"',
                    'Wheeler, Mr. Edwin Frederick""',
                    'Willer, Mr. Aaron (Abi Weller")"'
        ]
}

df1 = pd.DataFrame.from_dict(data)
df2 = pd.DataFrame.from_dict(data)
df3 = pd.DataFrame.from_dict(data)

df1.Name = df1.Name.replace('"', '', regex = True)
df2.Name = df2.Name.replace('"', '', regex = False)
df3.Name = df3.Name.str.replace('"', '')

print("df1 equals df2?:", df1.equals(df2))
print("df1 equals df3?:", df1.equals(df3))
print(set(df1.Name) - set(df2.Name))
print(set(df1.Name) - set(df3.Name))

Output:

df1 equals df2?: False
df1 equals df3?: True
{'Moubarek, Mrs. George (Omine Amenia Alexander)', 'McCarthy, Miss. Catherine Katie', 'Cotterill, Mr. Henry Harry', 'Katavelas, Mr. Vassilios (Catavelas Vassilios)', 'Coutts, Mrs. William (Winnie Minnie Treanor)', 'Hocking, Miss. Ellen Nellie', 'Wheeler, Mr. Edwin Frederick', 'Thomas, Mrs. Alexander (Thamine Thelma)', 'Johnston, Mrs. Andrew G (Elizabeth Lily Watson)', 'Dean, Miss. Elizabeth Gladys Millvina', 'Willer, Mr. Aaron (Abi Weller)', 'Nourney, Mr. Alfred (Baron von Drachstedt)', 'Wells, Mrs. Arthur Henry (Addie Dart Trevaskis)', 'Assaf Khalil, Mrs. Mariana (Miriam)', 'Daly, Miss. Margaret Marcella Maggie', 'Johnston, Master. William Arthur Willie', 'Riihivouri, Miss. Susanna Juhantytar Sanni', 'Rosenshine, Mr. George (Mr George Thorne)', 'Nakid, Mrs. Said (Waika Mary Mowad)', 'Riordan, Miss. Johanna Hannah', 'Lindeberg-Lind, Mr. Erik Gustaf (Mr Edward Lingrey)', 'Khalil, Mrs. Betros (Zahie Maria Elias)'}
set()
user_13552870
Related