Drop rows based on column values in Dask

user_17595239 0
Viewed 319

I am using dask to read a large csv file. I wanted to drop a few rows based on one column value. If for that particular column the row value is empty I want to remove the entire row.

I tried using .dropna:

df = df.dropna(subset=['tier1_name'],how = 'any',axis =0)

However, I got this error:

TypeError: dropna() got an unexpected keyword argument 'axis'

So I used .drop instead:

df.drop(df['tier1_name'].isnull(), axis = 0)

But then got this error:

"Drop currently only works for axis=1 or when columns is not None"
NotImplementedError: Drop currently only works for axis=1 or when columns is not None

I don't understand what should I use to execute the desired operation. Help!

drop dask rows null python
1 Answers

The key issue here is that, in general, dask will not know the number of rows or their content without evaluation, so row-based operations are not always easy to integrate.

As one solution to this, it's possible to use .loc with an appropriate mask, this pseudo-code might help:

mask = df['tier1_name'].notna()
df_modified = df.loc[mask]
# note that if you need to use .isna(), then the mask value
# should be negated
user_106935960
Related