How to write a code for a dataset in which one of the columns contains punctuations, spaces and to delete the corresponding row to it?

user_11696775 0
Viewed 57

I am trying to clean a dataset which contain some chinese characters and get rid of those rows which contain chinese characters.

I have first tried replacing chinese characters with a space and then tried to use regex to fill the dataset with only those rows and columns which don't have the spaces and punctuations.

        df["reviewer_name"] = df["reviewer_name"].str.replace(r'[^\x00-\x7F]+','')
        df['comments'] = df['comments'].str.replace(r'[^\x00-\x7F]+', '')
        df = df[df['comments'].str.contains(r'\W+', na=False)]
        df

The data is like this -

  • data -
    title_id date Reviewer_name comments

    258716 2019-04-21 Heap Chuan 新公寓,很干净,还有管理员接待

-Expected- All the rows with chinese character to be gone from dataset

pandas data-cleaning dataframe nlp python
1 Answers

Find rows containing the regular expression: rows_to_drop are the indicies of rows containing chinese.

rows_to_drop1= df.loc[df["reviewer_name"].str.contains(r'[^\x00-\x7F]', na=False)].index
clean_df = df.drop(rows_to_drop1,axis=0)
rows_to_drop2= df.loc[df["comments"].str.contains(r'[^\x00-\x7F]', na=False)].index

Now you want to update clean_df by dropping rows_to_drop2, so set inplace=True:

clean_df.drop(rows_to_drop2,axis=0,inplace=True)

You can also do it all at once:

condition1 = df["reviewer_name"].str.contains(r'[^\x00-\x7F]', na=False)
condition2 = df["comments"].str.contains(r'[^\x00-\x7F]', na=False)
row_to_drop = df.loc[condition1 & condition2 ].index
clean_df = df.drop(rows_to_drop,axis=0)
user_116665600
Related