I have a requirement where I need to move a specific file from an SFTP Server to an S3 Bucket. I am currently using the below code to move the required file if I provide its complete path(including filename). There are multiple files in the SFTP directory, but I only want to move files which are of .xlsx format or has .xlsx in filename, please suggest how I can do that. I am using SFTPtoS3Operator to get the file.
import pysftp
from airflow.models import DAG
from airflow.operators.python import PythonOperator
from airflow.providers.sftp.operators.sftp import SFTPOperator
from airflow.providers.sftp.sensors.sftp import SFTPSensor
from airflow.utils.dates import days_ago
from airflow.models import Variable
from airflow import models
from airflow.providers.amazon.aws.transfers.sftp_to_s3 import SFTPToS3Operator
with DAG("sftp_operators_workflow",
schedule_interval=None,
start_date=days_ago(1)) as dag:
wait_for_input_file = SFTPSensor(task_id="check-for-file",
sftp_conn_id="ssh_conn_id",
path="<full path with filename>",
poke_interval=10)
sftp_to_s3 = SFTPToS3Operator(
task_id="sftp_to_s3",
sftp_conn_id="ssh_conn_id",
sftp_path="<full path with filename>",
s3_conn_id="s3_conn_id",
s3_bucket="<bucket name>",
s3_key="full bucket path with filename")
wait_for_input_file >> sftp_to_s3
The file that needs to be moved as the filename like : M.DD.YYYY.xlsx. I really appreciate all the help & support on this one.