dask read_csv set the default value when the expected column is not available

sridharnetha 0
Viewed 28

How to set the default value on expected column but not found on csv? because my csv file headers are not consistence. I want to add column with the default value.

ValueError: Usecols do not match columns, columns expected but not found: ['Gender']

Here is my dask/pandas read_csv call,

df=dd.read_csv(filepath, low_memory=False, 
usecols=cols,
#usecols=lambda c: c in set(cols),
sep = ",", header = 0, encoding = "ISO-8859-1",
converters=_converters    
)
dask pandas python
1 Answers

You can’t currently do this with the dask.dataframe.read_csv reader. You could read each file in using dask.delayed and then convert to a dataframe using dask.dataframe.from_delayed, e.g.:

def read_one_file(fp, cols):
    df=pd.read_csv(
        fp,
        low_memory=False,
        sep = ",",
        header = 0,
        encoding = "ISO-8859-1",
        converters=_converters    
    ).reindex(columns=cols)
    return df

df = dask.dataframe.from_delayed(
    map(
        delayed(read_one_file),
        filepaths,
        cols=cols,
    )
)

Once this is done you can then fill missing values however you’d like.

Michael Delgado0
Related