In Airflow most pipeline consists of a group of DAGs each DAG contains tightly related tasks. Does Airflow have some construct/object which can group DAGs together? So that one can track the success or failure of the whole pipeline?
Something like in the picture:
To create a relationship between two separate dags, airflow provides ExternalTaskSensor to add a dependency to a task in another dag. When you use this sensor, you can access the graph of DAG dependencies available on <AIRFLOW_SERVER_URI>/dag-dependencies.
Also, Airflow provide an experimental lineage support based on OpenLineage, which allows you to use an external lineage platforms (ex: Marquez). By using a lineage tool, you can define the dependencies between the different airfow dags to create bigger business/functional dags, and track the success or failure of the whole pipeline.
