Currently we have a requirement to run spark jobs on Dataproc via airflow.
We are able to run Spark Jobs on Dataproc via airflow and able to check logs on GCP Console.
But wondering if there is a way to capture Dataproc job id from airflow and download Dataproc Job logs to GCS bucket ?
Dataproc operators use operator extra link this means that in the task actions you have a link to the dataproc resources:
If you still want to get the id then you can create a custom operator that push this value to Xcom (or does whatever you want with it). For example in DataprocInstantiateInlineWorkflowTemplateOperator you will need to return the workflow_id value in the execute function.
