AWSAathenaOperator output S3 name?

user_29734 0
Viewed 209

I'm using Airflow 2 and trying to use the AWSAthenaOperator. I can run the operator and it works, but I can't find any way to determine what the file names are that it wrote.

        task = AWSAthenaOperator(
            task_id="foo",
            database="mydb",
            query='select * from mytable limit 10;',
            aws_conn_id="athena_conn",
            output_location='s3://mybucket/myfolder',
        )

It drops files in s3://mybucket/myfolder, which is great, but how do I find out from the task output what those file names are? I need to then take those names and pass them to other tasks downstream.

I have been digging through the AWSAthenaOperator and AWSAthenaHook code that seems to do the work underneath, but I can't find where it stores that information or how I'd retrieve it.

airflow
1 Answers

I think the problem is that the path that you pass with output_location isn't unique. output_location is a templated field so you can use execution_date to make it unique thus each task save the file to a different and known location so files from different tasks are not getting mixed also using that way you can use the path in downstream tasks.

    task = AWSAthenaOperator(
        task_id="foo",
        database="mydb",
        query='select * from mytable limit 10;',
        aws_conn_id="athena_conn",
        output_location='s3://mybucket/myfolder/{{ execution_date }}',
    )
user_146244090
Related