Convert Parameter to Artifact without intermediary step

Question

I would like to iterate through a JSON artifact using withParams. Currently, I have to use an intermediary template, called get-json, as shown below:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
spec:
  arguments:
    parameters:
      - name: s3-bucket
      - name: input-json
      - name: output-path

  volumes:
    - name: workdir
      emptyDir: {}

  entrypoint: main

  templates:
    - name: main
      steps:
        - - name: get-inputs
            template: get-json
            arguments:
              artifacts:
                - name: json-file
                  s3:
                    endpoint: s3.amazonaws.com
                    bucket: "{{workflow.parameters.s3-bucket}}"
                    key: "{{workflow.parameters.input-json}}"

        - - name: iter-json
            # Iterates over the a set of work units produced by the previous step
            template: big-compute
            arguments:
              parameters:
                - name: param0
                  value: "{{item.param0}}"
                - name: param1
                  value: "{{item.param1}}"
                - name: param2
                  value: "{{item.param2}}"
            withParam: "{{steps.get-inputs.outputs.parameters.json-data}}"

    - name: get-json
      inputs:
        artifacts:
          - name: json-file
            path: /tmp/input.json
      outputs:
        parameters:
          - name: json-data
            valueFrom:
              path: /tmp/data.json
      script:
        image: alpine:latest
        command: [sh]
        source: |
          cp {{inputs.artifacts.json-file.path}} {{outputs.parameters.json-data.path}}
  
    - name: big-compute
      inputs:
        parameters:
          - name: param0
          - name: param1
          - name: param2
      outputs:
        artifacts:
          - name: collected-outputs
            path: /workdir/out
            archive:
              none: {}
            s3:
              endpoint: s3.amazonaws.com
              bucket: "{{workflow.parameters.s3-bucket}}"
              key: "{{workflow.parameters.output-path}}"
      script:
        image: fedora:33
        command: [bash]
        source: |
          set -xe

          mkdir /workdir/out && cd /workdir/out
          echo "{{inputs.parameters.param0}},{{inputs.parameters.param1}},{{inputs.parameters.param2}}" \
                > {{inputs.parameters.param0}}_{{inputs.parameters.param1}}_{{inputs.parameters.param2}}.txt

        resources:
          requests:
            memory: 30Mi
            cpu: 20m
          limits:
            memory: 30Mi
            cpu: 20m
        volumeMounts:
          - name: workdir
            mountPath: /workdir

Is there any way to iterate over a JSON artifact using withParams without this conversion?

user_548983 · Answer

I think want you want is this:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: artifact-passing-
spec:
  entrypoint: artifact-example
  templates:
  - name: artifact-example
    steps:
    - - name: process-var
        template: process-var
    - - name: print-artifact
        template: print-message
        arguments:
          parameters:
          - name: file
            value: "{{steps.process-var.outputs.result}}"

  - name: process-var
    inputs:
      parameters:
        - name: test
          value: "[\"foo\"]"
    data:
      transformation:
        - map:
            expression: "{{item}} + \"-bar\""

  - name: print-message
    inputs:
      parameters:
      - name: file
    container:
      image: alpine:latest
      command: [sh, -c]
      args: ["echo {{inputs.parameters.file}}"]

Credits: https://github.com/argoproj/argo-workflows/issues/5069

Convert Parameter to Artifact without intermediary step

1 Answers