creating boto3 s3 client on Airflow with an s3 connection and s3 hook

user_2192889 0
Viewed 3999

I am trying to move my python code to Airflow. I have the following code snippet:

    s3_client = boto3.client('s3', 
                        region_name="us-west-2", 
                        aws_access_key_id=aws_access_key_id,
                        aws_secret_access_key=aws_secret_access_key)

I am trying to recreate this s3_client using Aiflow's s3 hook and s3 connection but cant find a way to do it in any documentation without specifying the aws_access_key_id and the aws_secret_access_key directly in code.

Any help would be appreciated

airflow amazon-s3
1 Answers

You need to define aws connection in Admin -> Connections or with cli (see docs). Once the connection defined you can use it in S3Hook. Your connection object can be set as:

Conn Id: <your_choice_of_conn_id_name>
Conn Type: Amazon Web Services
Login: <aws_access_key>
Password: <aws_secret_key>
Extra: {"region_name": "us-west-2"}

enter image description here

In Airflow the hooks wrap a python package. Thus if your code uses hook there shouldn't be a reason to import boto3 directly.

user_146244090
Related