Can AWS Glue Crawler handle different file types in same folder?

user_10345434 0
Viewed 1283

I have reports delivered to S3 in the following structure:

s3://chum-bucket/YYYY/MM/DD/UsageReportYYYYMMDD.zip
s3://chum-bucket/YYYY/MM/DD/SearchReportYYYYMMDD.zip
s3://chum-bucket/YYYY/MM/DD/TimingReportYYYYMMDD.zip

The YYYY MM DD vary per day. The YYYMMDD in the filename is there because the files all go into one directory on a server before they are moved to S3.

I want to have 1 or 3 crawlers that deliver 3 tables to the catalog, one for each type of report. Is this possible? I can't seem to specify

s3://chum-bucket/**/UsageReport*.zip
s3://chum-bucket/**/SearchReport*.zip
s3://chum-bucket/**/TimingReport*.zip

I can write one crawler that excludes SearchReport and TimingReport, and therefore crawls the UsageReport only. Is that the best way?

Or do I have to completely re-do the bucket / folder / file name design?

aws-glue-data-catalog amazon-redshift aws-glue amazon-web-services
1 Answers

Amazon Redshift loads all files in a given path, regardless of filename.

Redshift will not take advantage of partitions (Redshift Spectrum will, but not a normal Redshift COPY statement), but it will read files from any subdirectories within the given path.

Therefore, if you want to load the data into separate tables (UsageReport, SearchReport, TimingReport), the they need to be in separate paths (directories). All files within the designated directory hierarchy must be in the same format and will be loaded into the same table via the COPY command.

An alternative is that you could point to a specific file using manifest files, but this can get messy.

Bottom line: Move the files to separate directories.

user_1747770
Related