Apache airflow cannot locate AWS credentials when using boto3 inside a DAG

Viewed 2635

We are migrating to Apache Airflow using ECS Fargate.

The problem we are facing, it's simple. We have a simple DAG that one of its tasks is to communicate with some external service in AWS (let's say, download a file from S3). This is the script of the DAG:

from airflow import DAG
from airflow.operators.bash_operator import BashOperator
from airflow.operators.python_operator import PythonOperator

from datetime import datetime, timedelta


# default arguments for each task
default_args = {
    'owner': 'thomas',
    'depends_on_past': False,
    'start_date': datetime(2015, 6, 1),
    'retries': 1,
    'retry_delay': timedelta(minutes=1),
}


dag = DAG('test_s3_download',
          default_args=default_args,
          schedule_interval=None) 

TEST_BUCKET = 'bucket-dev'
TEST_KEY = 'BlueMetric/dms.json'


# simple download task
def download_file(bucket, key):
    import boto3
    s3 = boto3.resource('s3')
    print(s3.Object(bucket, key).get()['Body'].read())


download_from_s3 = PythonOperator(
    task_id='download_from_s3',
    python_callable=download_file,
    op_kwargs={'bucket': TEST_BUCKET, 'key': TEST_KEY},
    dag=dag)


sleep_task = BashOperator(
    task_id='sleep_for_1',
    bash_command='sleep 1',
    dag=dag)


download_from_s3.set_downstream(sleep_task)

As we have done other times when using docker, we create within the docker container, in ~/.aws the config file that reads:

[default]
region = eu-west-1

and as long as the container is within the AWS boundaries, it'll resolve every request without any need to specify credentials.

This is the Dockerfile we are using:

FROM puckel/docker-airflow:1.10.7

USER root

COPY entrypoint.sh /entrypoint.sh
COPY requirements.txt /requirements.txt

RUN apt-get update

RUN ["chmod", "+x", "/entrypoint.sh"]

RUN mkdir -p /home/airflow/.aws \
&& touch /home/airflow/.aws/config \
&& echo '[default]' > /home/airflow/.aws/config \
&& echo 'region = eu-west-1' >> /home/airflow/.aws/config

RUN ["chown", "-R", "airflow", "/home/airflow"]

USER airflow

ENTRYPOINT ["/entrypoint.sh"]

# # Expose webUI and flower respectively
EXPOSE 8080
EXPOSE 5555

and everything works like a charm. Directory and change of owner are done successfully but when running the DAG, it fails saying:

...
...
File "/usr/local/airflow/.local/lib/python3.7/site-packages/botocore/signers.py", line 160, in sign
    auth.add_auth(request)
  File "/usr/local/airflow/.local/lib/python3.7/site-packages/botocore/auth.py", line 357, in add_auth
    raise NoCredentialsError
botocore.exceptions.NoCredentialsError: Unable to locate credentials
[2020-08-24 11:15:02,125] {{taskinstance.py:1117}} INFO - All retries failed; marking task as FAILED

So we are thinking that the worker node of Airflow does use another user.

Does any of you know what's going on? Thank you for any advice/light you can provide.

1 Answers

Create a proper task_role_arn for the task definition. This role is the one assumed by the processes triggered inside the container. Another annotation is that the error should not read:

Unable to locate credentials

that misleads, but

Access Denied: you don't have permission to s3:GetObject.

Related