Is there a way to download latest files from S3 bucket into my local system using Airflow .
Since I am a newbie to Airflow I don't have much idea on how to proceed. Please assist.
1 Answers
Short answer: You could use S3KeySensor to detect when a certain key appears in an S3 bucket and then use S3Hook.read_key() to get the content of the key.
Assuming you are completely new to Airflow, I would suggest:
- Start with the tutorial
- Read up on Connections, Hooks, and Sensors
- Use this example as a starting point for your own DAG
As a followup:
- Browse the amazon provider package docs to see what else there is for working with AWS services
- Look through other examples
Related