AWS Glue Crawlers add partitions within exclude pattern conditions

user_11987119 0
Viewed 70

I ran into the following situation: let's say I have the following s3 structure

  • s3://my_bucket/path_to_crawl/partition=A/some_file.parquet
  • s3://my_bucket/path_to_crawl/partition=B/some_file.parquet
  • s3://my_bucket/path_to_crawl/partition=C/some_file.csv

I point my crawler to s3://my_bucket/path_to_crawl and it has the following exclude pattern defined: **.csv. The desired output would be a table path_to_crawl created with 2 partitions s3://my_bucket/path_to_crawl/partition=A/ and s3://my_bucket/path_to_crawl/partition=B/, however the crawler also adds partition s3://my_bucket/path_to_crawl/partition=C/. The crawler does not however create a separate table for the last partition which is desired as not having the exclude patter would create a separate table.

The problem here is that in table path_to_crawl partition s3://my_bucket/path_to_crawl/partition=C/ inherits schema from the table and when trying to query it through e.g. Athena - the query naturally errors out as it encounters incompatible schema.

Is there a way to make exclude patterns not add the partition to the table as well?

aws-glue-data-catalog aws-glue amazon-web-services
1 Answers

Is there a way to make exclude patterns not add the partition to the table as well?

You'll have to explicitly exclude the path of the partition.

Include path:

s3://my_bucket/path_to_crawl

Exclude paths:

**.csv
partition=C/*
user_14360310
Related