Pandas Dataframe filter not working but str.match() is working

user_3302140 0
Viewed 942

I have a Pandas Dataframe words_df which contains some English words.

It only has one column named word which contains the English word.

words_df.tail():

enter image description here

words_df.dtypes:

enter image description here

I want to filter out the row(s) which contain the word zythum

Using the Pandas Series str.match() is giving me expected output:

words_df[words_df.word.str.match('zythum')]:

I know str.match() is not the correct way to do it, it will also return rows which contain other words like zythums for example.

enter image description here

But, using the following operation on Pandas Dataframe is returning an empty Dataframe

words_df[words_df['word'] == 'zythum']:

enter image description here

I was wondering why is this happening?

EDIT 1: I am also attaching the source of my data and the code used to import it.

Data source (I used "Word lists in csv.zip"):

https://www.bragitoff.com/2016/03/english-dictionary-in-csv-format/

Dataframe import code:

import pandas as pd
import glob as glob
import os as os
import csv

path = r'data/words/' # use your path
all_files = glob.glob(path + "*.csv")

li = []

for filename in all_files:
    df = pd.read_csv(filename, index_col=None, header=None, names = ['word'], engine='python', quoting=csv.QUOTE_NONE)
    li.append(df)

words_df = pd.concat(li, axis=0, ignore_index=True)

EDIT 2:

Here is a block of my code, with a simpler import code, but facing same issue. (using Zword.csv file from link mentioned above) enter image description here

pandas dataframe python
4 Answers

IIUC: df1[df1['word'] == 'zythum'] is not working.

Try, removing whitespace around the string in the dataframe:

 df1[df1['word'].str.strip() == 'zythum']
user_63615310

Your imported list does not match the string you are looking for exactly. There is a space after the words in the csv file.

You should be able to strip the whitespace out by using str.strip. For example:

import pandas as pd
myDF = pd.read_csv('Zword.csv')
myDF[myDF['z '] == 'zythum '] # This has the whitespace
myDF['z '] = myDF['z '].map(str.strip)
myDF[myDF['z '] == 'zythum'] # mapped the whitespace away
user_136076080

You need to convert the whole column to str type:

    words_df['word'] = words_df['word'].astype(str)

This should work in your case.

user_84389820

Here, you can use this to do the work. Change parameters as required.

import glob as glob
import os as os
import csv

def match(dataframe):
    l = []
    for i in dataframe:
        l.append('zythum' in i)
    data = pd.DataFrame(l)
    data.columns = ['word']
    return data

path = r'Word lists in csv/' # use your path
files = os.listdir(path)
li = []
for filename in files:
    df = pd.read_csv(path + filename, index_col=None, header=None, names = ['word'], engine='python', quoting=csv.QUOTE_NONE)
    li.append(df)
words_df = pd.concat(li, axis=0, ignore_index=True)

words_df[match(words_df['word'])].dropna()```
user_117806250
Related