Use run parameter as arg. kubeflow

Viewed 269

I am trying to use a kubeflow run parameter as an argument for my pipeline step.

Every time I compile the yaml file however it gets changed from an Integer to a LocalPath.

@dsl.pipeline(name='First Pipeline', description='generates a random set of numbers then performs operations on them returning a json object')
def first_pipeline(generate_n_arg: int = 10):

    # Loads the yaml manifest for each component
    generate_numbers = kfp.components.load_component_from_file('generate_numbers/generate_numbers.yaml')

    multiply_numbers = kfp.components.load_component_from_file('multiply/multiply.yaml')
    sum_numbers = kfp.components.load_component_from_file('sum/sum.yaml')
    sort_numbers = kfp.components.load_component_from_file('sort/sort.yaml')
    
    output_to_json = kfp.components.load_component_from_file('output_to_json/output_to_json.yaml')

    # Run generate_data task
    print(generate_n_arg)
    generate_numbers_task = generate_numbers(generate_n=generate_n_arg)

my yaml file clearly dictates that it needs to be an integer

description: generate random toy data

inputs:
- {name: generate_n, type: Integer, description: 'an integer for how many numbers to generate'}

outputs:
- {name: out_path, type: LocalPath, description: 'Path where data will be stored.'}

implementation:
  container:
    image: path/to/image
    command: [
      python, generate.py,

      --generate_n,
      {inputPath: generate_n},

      --out_path,
      {outputPath: out_path},
    ]

however every time I compile the pipeline it gets changed to a path

command: [python, generate.py, --generate_n, /tmp/inputs/generate_n/data, --out_path,
        /tmp/outputs/out_path/data]

even though the pipeline still acknowledges that I input an integer multiple times in the yaml

    inputs:
      artifacts:
      - {name: generate_n_arg, path: /tmp/inputs/generate_n/data}

  arguments:
    parameters:
    - {name: generate_n_arg, value: '10'}
    artifacts:
    - name: generate_n_arg
      raw: {data: '{{workflow.parameters.generate_n_arg}}'}

Does anyone know how I can make it give me the input value instead of an localpath/artifact path?

1 Answers

I found the problem. I had an incorrect value in my yaml file. I was using the wrong input command to give a value.

changing

      --generate_n,
      {inputPath: generate_n},

to

      --generate_n,
      {inputValue: generate_n},
Related