Removing rows from the dataframe 1 where equivalent row in dataframe2 has null values

user_8489816 0
Viewed 73

I have two data frames and I am using the last columns from both the data frames to calculate classification metrics.

I want to make sure the values are correct, one of my data-frames is being loaded from database and there are chances that I might get NaN values in that data frame.

I want to delete the row which consist of NaN values from both the data frames, my other dataframe won't have any NaN but I want to check NaN values in first data frame and delete the equivalent row from first and second data frame so that my classification happens only on real values.

I am not sure how to achieve this

2 Datagframes

pandas dataframe python
3 Answers

You can use df.isna along with df.any to create a boolean mask, then you can filter the dataframes using this mask:

m = df2.isna().any(axis=1) # boolean mask representing rows in df2 having nan values
df1 = df1[~m]
df2 = df2[~m]

This results:

# print(df1)
   product_id   style   label   store
0         100  style0  label0  store0
1         101  style1  label1  store1
3         103  style3  label3  store3

# print(df2)
   product_id   style   label   store
0         100  style0  label0  store0
1         101  style1  label1  store1
3         103  style3  label3  store3
user_128331660

There are several ways of approaching this problem, the way I would get to it is by first getting all the index of the null values in the df1 and than find that index in df2 and remove it:

   rows_with_nans = [index for index, row in df1.iterrows() if row.isnull().any()]
   print(rows_with_nans)

And in order to delete from df2 I would do this :

  df2.drop(rows_with_nans)
user_15253960 
mask = df_database.isna()
df_database.drop(mask, axis=0, inplace=True)
df_non_databse.drop(mask, axis=0, inplace=True)
user_135221870
Related