How to make a kubeflow pipeline step depend on multiple previous steps

G. Macia 0
Viewed 35

I am running several kf steps in parallel. When they all complete AND if they have all succeeded I would like to trigger a last final step. With my current implementation the last step triggers if any of the previous ones succeeds which is not what I intend.

I have been looking at the documentation but I could not find a straightforward way to do it. Could someone provide an example?

kubeflow-pipelines kubeflow
2 Answers

If kubeflow knows about a dependency between your steps (eg. step B depends on the output of step A) then this will happen automatically. Otherwise, you can just do this

from kfp import components, dsl, Client

@components.func_to_container_op
def echo(text: str)-> str:
    print(text)
    return text

@dsl.pipeline("with-afters")
def my_pipeline():
    parallel_1 = echo("bish")
    parallel_2 = echo("bash")

    serial_1 = echo("bosh").after(parallel_1).after(parallel_2)

If you want to wait for a loop it's easy too:

@dsl.pipeline("with-loop")
def loop_pipeline():
    with dsl.ParallelFor(["bish", "bash"]) as word:
        loop_step = echo(word)

    serial_1 = echo("bosh").after(loop_step)
Tom Clelford0

You should return a Boolean output from each of them and then check whether they're all true in the downstream step. I don't know Kubeflow syntax off the top of my head but with ZenML (which you run on Kubeflow) it would look like:

@step
def step1() -> bool:
   if successful:
       return True
   return False

@step
def step2() -> bool:
   if successful:
       return True
   return False

@step
def step3() -> bool:
   if successful:
       return True
   return False

@step
def downstream(step1: book, step2: book, step3: bool) -> bool:
   if step1 and step2 and step3:
       # execute stuff
       return True
   return False

@pipeline
def p(step1, step2, step3, downstream):
   downstream(
      step1(),
      step2(),
      step3(),
   )

p(step1(), step2(), step3(), downstream ()).run()
Hamza Tahir0
Related