Using Airflows S3Hook is there a way to copy objects between buckets with different connection ids?

Viewed 11

I'm copying files from an external companies bucket, they've sent me an access key/secret that I've set up as an env variable. I want to be able to copy objects from their bucket, I've used the below but that's for moving objects with the same connection, how do I use S3Hook to copy objects w. a different conn id?

    s3 = S3Hook(self.aws_conn_id)
    s3_conn = s3.get_conn()

    ext_s3 = S3Hook(self.ext_aws_conn_id)
    ext_s3 conn = ext_s3.get_conn()

    #this moves objects w. the same connection...
    s3_conn.copy_object(Bucket="bucket",
                        Key=f'dest_key',
                        CopySource={
                            'Bucket': self.partition.bucket,
                            'Key': key
                            }, ContentEncoding='csv')
0 Answers
Related