Running a kedro pipeline with inputs and outputs defined through the command line

user_16876549 0
Viewed 626

I would like to run a kedro pipeline using different inputs and saving the results in an output folder where inputs paths and outputs paths are provided through the command line

I sow the possibility of using the kedro.config.TemplatedConfigLoader to pass new variables to a jinja2 template catalog, but in this way I can only manually define the globals_dict variables in the hooks as shown in the kedro documentation.

Ideally I would like to have to run something like this:

kedro run --pipeline="my_pipeline" --input="path_to_input_1" --output="path_to_output_1"
kedro run --pipeline="my_pipeline" --input="path_to_input_2" --output="path_to_output_2"

with a catalog like this:


input_df:
  type: pandas.CSVDataSet
  filepath: "${ input_path }"
  load_args:
    sep: "\t"
    index_col: 0
  save_args:
    index: True
    encoding: "utf-8"

output_df:
  type: pandas.CSVDataSet
  filepath: "${ output_path }"
  load_args:
    sep: "\t"
    index_col: 0
  save_args:
    index: True
    encoding: "utf-8"

and having the correct inputs analysed and the results stored in the correct output paths.

what would be the kedro way to achieve it?

kedro command-line
1 Answers

so this isn't available by default. You could try experimenting with a few parts of kedro to make this work:

  • Edit cli.py to provide your new Click arguments
  • Experiment with before_dataset_loaded hooks to add your arguments to the globals.yml dictionary which TemplatedConfigLoader uses for string interpolation

That being said, perhaps run environments may be helpful for this?

https://kedro.readthedocs.io/en/latest/04_kedro_project_setup/02_configuration.html#additional-configuration-environments

In general we advise against dynamic pipelines which are hard to read at rest. We do encourage people to keep their catalogs explicit so they are readable in 6 months time.

user_20108080
Related