DatabricksRunNowOperator returning value

Viewed 267

Can I pass a return value from DatabricksRunNowOperator using xcom or any other method ? I just want to return back "date" after my 1st databricks operator has finished and pass it on to the dependent task

for eg:

I want to pass the return value of the verification_run to the insert_run and workspace_run. Usually we can use a xcom_pull and xcom_push to do it in python. But not sure how to make two notebooks talk to each other

from airflow import DAG
from airflow.providers.databricks.operators.databricks import DatabricksRunNowOperator
from airflow.utils.dates import days_ago

default_args = {
  'owner': 'airflow'
}

with DAG('databricks_dag',
  start_date = days_ago(2),
  schedule_interval = None,
  default_args = default_args
  ) as dag:

  verification_run = DatabricksRunNowOperator(
    task_id = 'verification_task',
    databricks_conn_id = 'databricks_default',
    job_id = '-----'
  )

  insert_run = DatabricksRunNowOperator(
    task_id = 'insert_task',
    databricks_conn_id = 'databricks_default',
    job_id = '-----'
  )
  workspace_run = DatabricksRunNowOperator(
    task_id = 'workspace_task',
    databricks_conn_id = 'databricks_default',
    job_id = '------'
  )

  verification_run >> [insert_run,workspace_run]




0 Answers
Related