I read that the Glue Data Catalog needs a Crawler to run to see any new partitions, or to use the new enableUpdateCatalog feature for AWS Glue ETL.
If, however,
- running native Spark Jobs on EMR and having selected Glue data Catalog as the Spark persistent MetaStore, then,
- I am assuming that if you have added a partition to a Spark table,
- that AWS Glue Data Catalog will be updated automatically.
- I am assuming that if you have added a partition to a Spark table,
That is what I am seeking to confirm. Would seem odd if not, but...
From the manual:
AWS Glue provides a number of ways to populate metadata into the AWS AWS Glue Data Catalog. AWS Glue crawlers scan various data stores you own to automatically infer schemas and partition structure and populate the AWS AWS Glue Data Catalog with corresponding table definitions and statistics. You can also schedule crawlers to run periodically so that your metadata is always up-to-date and in-sync with the underlying data. Alternately, you can add and update table details manually by using the AWS Glue Console or by calling the API. You can also run Hive DDL statements via the Amazon Athena Console or a Hive client on an Amazon EMR cluster. Finally, if you already have a persistent Apache Hive Metastore, you can perform a bulk import of that metadata into the AWS AWS Glue Data Catalog by using our import script.
I am assuming that Spark on EMR will use that API.