Unable to access data in tables configured using Partition Projection from AWS glue jobs using create_dynamic_frame.from_catalog

user_214518 0
Viewed 189

I've setup a table in Athena using partition projection. I haven't defined any partitions in the glue metadata catalogue and I can view the data in Athena OK using SQL.

When I go and setup a Glue job using this table, Glue doesn't seem to have access to the data:

data = glueContext.create_dynamic_frame.from_catalog(
    database="db", table_name="a_table")

print (data.count()) # returns 0 :(

Is there any way to access the data without needing to define Glue metadata partitions? I was under the impression that if Athena could see the data, so could Glue.

amazon-athena aws-glue amazon-web-services
1 Answers

Glue does not support Partition Projection, it's an Athena-only feature.

Glue ETL uses Spark, and Athena is Presto under the hood (with modifications, including Partition Projection). Glue ETL also does not support Athena views, and various other minor things.

user_11090
Related