Cloud Data Fusion triggered pipeline - reuse the already provisioned Dataproc clusters

Viewed 116

Is there a way to avoid the provisioning step for subsequently triggered outbound pipelines? It looks like when a pipeline triggers an outbound pipeline, it does the provisioning all over again. Can we simply execute the triggered pipeline on the provisioned cluster of the first?

Thanks.

1 Answers

Cloud Data Fusion has support for running a pipeline against an existing Dataproc cluster, see more details in this doc.

Related