tried below code but getting error as "unable to locate Credentials" def download():
bucket = 'bucketname'
key = 'path and filename'
s3_resource = boto3.resource('s3')
my_bucket = s3_resource.Bucket(bucket)
objects = my_bucket.objects.filter(Prefix=key)
for obj = objects:
path,filename = os.path.split(obj.key)
my_bucket.download_file(obj.key, filename)
Your code is Python with Airflow you don't need to interact with boto3 directly. You can use the S3Hook for that.
You'll need to define the AWS connection and use
download_fileobj function of the S3Hook.
I didn't test it but it should be something like:
from tempfile import NamedTemporaryFile
from airflow.providers.amazon.aws.hooks.s3 import S3Hook
hook = S3Hook('my_aws_conn')
source_s3_key_object = hook.get_key('your_path')
with NamedTemporaryFile("wb") as f:
hook.download_fileobj(Fileobj=f)
f.flush()