I started building a data processing pipeline with Apache Airflow and stuck with (possibly)
very basic problem: where to store files downloaded by operator1 and then has to be processed by operator2?
operator1 - BashOperator to download zip file from an external resource. I do this with curl to be able to retry or restart downloading because external resources are flaky.
operator2 - airflow.contrib.operators.file_to_wasb operator to upload to Azure Blob
So the question is how operator2 can access files downloaded by operator1?
Should operator1 upload files somewhere or it is possible to delegate this to next task?
operator1 = BashOperator(
task_id='download_ds_archive',
bash_command="curl --retry 10 -O -C - https://extresource/file.zip",
xcom_push=True,
provide_context=True,
dag=dag,
)
operator2 = FileToWasbOperator(
task_id='stage_xml_files_to_azure_blob',
...
)
operator1 >> operator2