Can I use Athena View as a source for a AWS Glue Job?

user_8978168 0
Viewed 4271

I'm trying to use an Athena View as a data source to my AWS Glue Job. The error message I'm getting while trying to run the Glue job is about the classification of the view. What can I define it as? Thank you

Error Message Appearing

amazon-athena aws-glue jobs amazon-web-services
1 Answers

You can by using the Athena JDBC driver. This approach circumvents the catalog, as only Athena (and not Glue as of 25-Jan-2019) can directly access views.

  1. Download the driver and store the jar to an S3 bucket.
  2. Specify the S3 path to the driver as a dependent jar in your job definition.
  3. Load the data into a dynamic frame using the code below (using an IAM user with permission to run Athena queries).
from awsglue.dynamicframe import DynamicFrame
# ...
athena_view_dataframe = (
    glueContext.read.format("jdbc")
    .option("user", "[IAM user access key]")
    .option("password", "[IAM user secret access key]")
    .option("driver", "com.simba.athena.jdbc.Driver")
    .option("url", "jdbc:awsathena://athena.us-east-1.amazonaws.com:443")
    .option("dbtable", "my_database.my_athena_view")
    .option("S3OutputLocation","s3://bucket/temp/folder") # CSVs/metadata dumped here on load
    .load()
    )

athena_view_datasource = DynamicFrame.fromDF(athena_view_dataframe, glueContext, "athena_view_source")

The driver docs (pdf) provide alternatives to IAM user auth (e.g. SAML, custom provider).

The main side effect to this approach is that loading causes the query results to be dumped in CSV format to the bucket specified with the S3OutputLocation key.

I don't believe that you can create a Glue Connection to Athena via JDBC because you can't specify an S3 path to the driver location.

Attribution: AWS support totally helped me get this working.

user_77110450
Related