I am relatively new to Airflow and am trying to solve a particular problem. I wanted to know how I could download my file residing in an S3 bucket after SSH to a remote machine/client through the SSHOperator. Can I simply use some bash script (I have tried this using the S3 URL, but it does not work)? or are there any operators available within airflow to perform this operation as part of my DAG? I tried the PythonOperator, but it seems to work only when I specify a local file path (in the op_kwargs dictionary). Below is my sample Airflow code; any help will be highly appreciated.
# command to execute on remote
download_file_cmd = 'rm -rf airflow_pipeline && mkdir -p airflow_pipeline && cd airflow_pipeline'
download_to_edge_node = SSHOperator(
task_id='download_file_to_edge_node',
ssh_conn_id='itversity',
command=download_file_cmd,
dag=dag
)
task_download_from_s3 = PythonOperator(
task_id='download_file_from_s3',
python_callable=download_from_s3,
op_kwargs={
'key': 'orders.csv',
'bucket_name': 'airflow-pipeline-files',
'local_path': '/home/danish/test'
}
)