I have 5 pipelines on my Azure Data Factory, each pipeline copy data to a different table. There is a dependency on some of this tables, table D & E depend on table A, B & C. Like in this example:
Table dependencies & Pipelines
What I'm doing to refresh all data is the following executing sequence:
Exec Pipeline to table A > Exec Pipeline to table B > Exec Pipeline to table C > Exec Pipeline to table D > Exec Pipeline to table E.
I could execute Pipeline to table E before Pipeline to table D with no problem, but none of them can be executed before Pipelines for table A, B & C.
The idea I had to make this more organized and easier to schedule was change the pipeline D and add there 3 steps that will execute the Pipelines for table A, B & C. And on Pipeline to table E I added a step to execute the pipeline D. Like in the example:
However, this would create a kind of dependency on Table E with Table D, which I don't want. If I need for any reason to update JUST table E, it won't be able because I would need to update table D first.
I wanted that both Pipelines to table D & E had a kind of validation if Pipelines to table A, B & C had run so then they can run.
Is there a way to make this dependencies more organized?
