How to trigger an Airflow dag from different GCP project?

user_5260122 0
Viewed 163

I have two independent DAGs let's say DAG_A and DAG_B, each has multiple tasks

The two DAGs are in different GCP projects let's say projct-1 and project-2 respectively.

  • What I want to do is to create a 3rd DAG let's call it DAG_C

  • DAG_C will be part of project-1 , and will be used to orchestrate DAG_A and DAG_B.

  • DAG_C should start by triggering DAG_A and on task_2 success it should trigger DAG_B

Please take a look at this picture that simplifies the problem: Overview of the architecture

The question is: would this be possible using the TriggerDagRunOperator , as I can't see any option to change the GCP project id on that operator ?

Also what would be the best approach to go towards that "assuming that TriggerDagRunOperator will not work in that case" ?

airflow-2.x airflow
1 Answers

There is no option to do that with TriggerDagRunOperator as the operator see only the scope of the Airflow instance that it's in.

Your only option is to use the Airflow Rest API. In DAG_C the trigger_B task will need to be a PythonOperator that authenticate with the Rest API of project_2 and then use the Trigger new DagRun endpoint to trigger DAG_B. Note that Airflow provides official Python client for the API so you can use it for this task.

user_146244090
Related