Trying to Drop values by column (I convert these values to nan but could be anything) not working

user_8484885 0
Viewed 39

Trying to drop NAs by column in Dask, given a certain threshold and I receive the error below.

I'm receiving the following error, but this should be working. Please advise.

enter image description here

reproducible example.

import pandas as pd
import dask

data = [['tom', 10], ['nick', 15], ['juli', 5]]
  
# Create the pandas DataFrame
df = pd.DataFrame(data, columns = ['Name', 'Age'])

import numpy as np
df = df.replace(5, np.nan)

ddf = dd.from_pandas(df, npartitions = 2)

ddf.dropna(axis='columns')
dask pandas
2 Answers

Passing axis is not support for dask dataframes as of now. You cvan also print docstring of the function via ddf.dropna? and it will tell you the same:

Signature: ddf.dropna(how='any', subset=None, thresh=None)
Docstring:
Remove missing values.

This docstring was copied from pandas.core.frame.DataFrame.dropna.

Some inconsistencies with the Dask version may exist.

See the :ref:`User Guide <missing_data>` for more on which values are
considered missing, and how to work with missing data.

Parameters
----------
axis : {0 or 'index', 1 or 'columns'}, default 0  (Not supported in Dask)
    Determine if rows or columns which contain missing values are
    removed.

    * 0, or 'index' : Drop rows which contain missing values.
    * 1, or 'columns' : Drop columns which contain missing value.

    .. versionchanged:: 1.0.0

       Pass tuple or list to drop on multiple axes.
       Only a single axis is allowed.

how : {'any', 'all'}, default 'any'
    Determine if row or column is removed from DataFrame, when we have
    at least one NA or all NA.

    * 'any' : If any NA values are present, drop that row or column.
    * 'all' : If all values are NA, drop that row or column.

thresh : int, optional
    Require that many non-NA values.
subset : array-like, optional
    Labels along other axis to consider, e.g. if you are dropping rows
    these would be a list of columns to include.
inplace : bool, default False  (Not supported in Dask)
    If True, do operation inplace and return None.

Returns
-------
DataFrame or None
    DataFrame with NA entries dropped from it or None if ``inplace=True``.

Worth noting that Dask Documentation is copied from pandas for many instances like this. But wherever it does, it specifically states that:

This docstring was copied from pandas.core.frame.DataFrame.drop. Some inconsistencies with the Dask version may exist.

Therefore its always best to check docstring for dask's pandas-driven functions instead of relying on documentation

user_72129290

The reason this isn't supported in dask is because it requires computing the entire dataframe in order for dask to know the shape of the result. This is very different from the row-wise case, where the number of columns and partitions won't change, so the operation can be scheduled without doing any work.

Dask does not allow some parts of the pandas API which seem like normal pandas operations which might be ported to dask, but in reality can't be scheduled without triggering compute on the current frame. You're running into this issue by design, because while .dropna(axis=0) would work just fine as a scheduled operation, .dropna(axis=1) would have a very different implication.

You can do this manually with the following:

ddf[ddf.columns[~ddf.isna().any(axis=0)]]

but the filtering operation ddf.columns[~ddf.isna().any(axis=0)] will trigger a compute on the whole dataframe. It probably makes sense to persist prior to running this if you can fit the dataframe in your cluster's memory.

user_38887190
Related