Using S3FileTransformOperator in MWAA enviroment

user_19209993 0
Viewed 119

I'm trying to use the S3FileTransformOperator in a MWAA enviroment, but I'm suffering from lack of permission in the script file:

PermissionError: [Errno 13] Permission denied

I tried to add a Bash operator with the chmod command before the task, but without success.

Has anyone ever used the S3FileTransformOperator in MWAA?

mwaa airflow amazon-s3 shell python
2 Answers
Hi!

Unfortunately with MWAA the worker containers are both ephemeral and limited to user level access. The S3 operator should work with .sh files added to the /dags folder and referred to in the operator as /usr/local/airflow/dags/my_script.sh. The alternative would be to use the contents of your .py file from a Python operator and use the S3Hook to retrieve and store the file.

Thanks!

Reference: https://repost.aws/questions/QUvsbZds_NQTG7JSxCQ11djQ/s-3-file-transform-operator-permission-denied-on-script

user_49614550

I tried this but got another error at first step,

from airflow.operators.bash import BashOperator
from airflow.operators.s3_file_transform_operator import S3FileTransformOperator

with DAG(...) as dag:
    chmod = BashOperator(
        task_id="chmod",
        bash_command="chmod +x /usr/local/airflow/dags/transform.py"
    )
    transform = S3FileTransformOperator(
        task_id="transform",
        source_s3_key="s3://bucket/path/to/sample.csv",
        dest_s3_key="s3://bucket/path/to/result.csv",
        transform_script="/usr/local/airflow/dags/transform.py",
    )

    chmod >> expose

chmod: changing permissions of ‘/usr/local/airflow/dags/transform.py’: Read-only file system
user_13604760
Related