Move specific file from SFTP to S3 using Airflow

Viewed 23

I have a requirement where I need to move a specific file from an SFTP Server to an S3 Bucket. I am currently using the below code to move the required file if I provide its complete path(including filename). There are multiple files in the SFTP directory, but I only want to move files which are of .xlsx format or has .xlsx in filename, please suggest how I can do that. I am using SFTPtoS3Operator to get the file.

import pysftp
from airflow.models import DAG
from airflow.operators.python import PythonOperator
from airflow.providers.sftp.operators.sftp import SFTPOperator
from airflow.providers.sftp.sensors.sftp import SFTPSensor
from airflow.utils.dates import days_ago
from airflow.models import Variable 
from airflow import models
from airflow.providers.amazon.aws.transfers.sftp_to_s3 import SFTPToS3Operator


with DAG("sftp_operators_workflow",
         schedule_interval=None,
         start_date=days_ago(1)) as dag:

    wait_for_input_file = SFTPSensor(task_id="check-for-file",
                                     sftp_conn_id="ssh_conn_id",
                                     path="<full path with filename>",
                                     poke_interval=10)
    
    
   
    sftp_to_s3 = SFTPToS3Operator(
                            task_id="sftp_to_s3",
                            sftp_conn_id="ssh_conn_id",
                            sftp_path="<full path with filename>",
                            s3_conn_id="s3_conn_id",
                            s3_bucket="<bucket name>",
                            s3_key="full bucket path with filename")



    wait_for_input_file >> sftp_to_s3

The file that needs to be moved as the filename like : M.DD.YYYY.xlsx. I really appreciate all the help & support on this one.

0 Answers
Related