Removing columns and rows from sparse dataset

user_13830093 0
Viewed 264

I have a sparse Pandas dataframe with many null values and I want to filter it such that only rows and columns with more than 10 float entries are retained in the final dataset. I have tried using an existing snippet of code but it doesn't seem to work:

df.drop([col for col, val = df.count(axis=1, numeric_only='float') if val < 10], axis=1, inplace=True)

Can anyone let me know what the best way to drop the sparse columns in my dataframe is?

pandas python
1 Answers

You can get the number of non-missing values in each row and column, check if it's greater than your threshold, then ask only for those rows/values where your condition is True.

kept_rows, kept_columns = df.isnull().sum(1)>10, df.isnull().sum(0)>10
df = df.loc[kept_rows, kept_columns]
user_138106260
Related