If I have a Dataproc workflow template scheduled using a Cloud Composer, Can I pass values to my dataproc jobs that are present inside Workflow Template through the Cloud Composer environment? If yes please mention the source. Thanks in advance!
1 Answers
If you use DataprocInstantiateWorkflowTemplateOperator operator then you can use parameters argument which is "a map of parameters for Dataproc Template".
The fact that the parameters argument is one of templated fields means it can be configured using for example:
- DAG run configuration via macros (using
{{ dag_run.conf }}) - Airflow variables (using for example
{{ var.value.get('my.var', 'fallback') }}) - Environment variables (
os.environ.get)
You can also hardcode this parameters if you don't need option to change them from time to time.
Related