How to iterate through a Pandas dataframe, and apply a threshold function to remove columns with x% as null?

user_14506353 0
Viewed 283

I am trying to drop columns according to the amount of NaN values they contain. The default threshold is 0.5, but should be able to be changed.

The code I have come up with so far is as follows:

def drop_cols_na(df, threshold=0.5):
    for column in df.columns:
        if df[column].isna().sum() / df.shape[0] >= threshold:
            df.drop([column], axis=1, inplace=True)
            return df

The code runs fine, but is having no effect and does not remove any columns. Any advice would be much appreciated! Thanks

pandas python
2 Answers

This function deletes columns having None values more than specified threshold.

sample_df = pd.DataFrame({'A':[2,3,None, 4,None], 'B':[2,13,None, None,None], 'C':[None,3,None, 4,None]})

def drop_cols_na(df, threshold = 0.5):
    df1 = df.copy()
    for i, col in df.iteritems():
        if((sum(col.isna())/len(col)) >= threshold):
            df1.drop(labels = i, axis = 1, inplace = True)
    return(df1)

# call the function with whatever threshold
drop_cols_na(sample_df, 0.5)
user_31725560

A simple way would be

threshold = 0.5
na_col = df.isna().sum().div(len(df.index))              # average na by column
drop_cols = na_col[na_col > threshold].keys().to_list()  # Find columns greater than threshold
df.drop(columns=drop_cols)                               # Drop columns
user_51200490
Related