kedro DataSetError while loading PartitionedDataSet

user_8030784 0
Viewed 381

I am using PartitionedDataSet to load multiple csv files from azure blob storage. I defined my data set in the datacatalog as below.

my_partitioned_data_set:
          type: PartitionedDataSet
          path: my/azure/folder/path
          credentials: my credentials
          dataset: pandas.CSVDataSet
          load_args:
                sep: ";"
                encoding: latin1

I also defined a node to combine all the partitions. But while loading each file as a CSVDataSet kedro is not considering the load_args, so I am getting the below error.

Failed while loading data from data set CSVDataSet(filepath=my/azure/folder/path, load_args={}, protocol=abfs, save_args={'index': False}).
'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte

The error shows that while loading the CSVDataSet kedro is not considering the load_args defined in the PartitionedDataSet. And passing an empty dict as a load_args parameter to CSVDataSet. I am following the documentation https://kedro.readthedocs.io/en/stable/05_data/02_kedro_io.html#partitioned-dataset I am not getting where I am doing mistakes.

kedro
1 Answers

Move load_args inside dataset

my_partitioned_data_set:
  type: PartitionedDataSet
  path: my/azure/folder/path
  credentials: my credentials
  dataset:
    type: pandas.CSVDataSet
    load_args:
      sep: ";"
      encoding: latin1

  • load_args mentioned outside dataset is passed into find() method of the corresponding filesystem implementation

  • To pass granular configuration to underlying dataset put it inside dataset as above.

You can check out the details in the docs

https://kedro.readthedocs.io/en/stable/05_data/02_kedro_io.html?highlight=partitoned%20dataset#partitioned-dataset-definition

user_46595300
Related