Can AWS Glue crawl Delta Lake table data?

user_4390959 0
Viewed 4443

According to the article by Databricks, it is possible to integrate delta lake with AWS Glue. However, I am not sure if it is possible to do it also outside of Databricks platform. Has someone done that? Also, is it possible to add Delta Lake related metadata using Glue crawlers?

delta-lake aws-glue apache-spark amazon-s3
4 Answers

This is not possible. Although you can crawl the S3 delta files outside the databrics platform but you won't find the data in the tables.

As per the doc, it says below :

Warning

Do not use AWS Glue Crawler on the location to define the table in AWS Glue. Delta Lake maintains files corresponding to multiple versions of the table, and querying all the files crawled by Glue will generate incorrect results.

user_31992850

I am currently using a solution to generate manifests of Delta tables using Apache Spark (https://docs.delta.io/latest/presto-integration.html#language-python).

I generate a manifest file for each Delta Table using:

deltaTable = DeltaTable.forPath(<path-to-delta-table>)
deltaTable.generate("symlink_format_manifest")

Then created the table using the example below. The DDL below also creates the table inside Glue Catalog; you can then access the data from AWS Glue using Glue Data Catalog.

CREATE EXTERNAL TABLE mytable ([(col_name1 col_datatype1, ...)])
[PARTITIONED BY (col_name2 col_datatype2, ...)]
ROW FORMAT SERDE 
'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'
STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.SymlinkTextInputFormat'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION '<path-to-delta-table>/_symlink_format_manifest/'  -- location of 
the generated manifest
user_46803340

It is finally possible to use AWS Glue Crawlers to detect and catalog Delta Tables.

Here is a blog post explaining how to do it.

user_126381180

It would be better if you could clarify what do you mean by saying "integrate delta lake with AWS Glue"..

At this moment, there is no direct Glue API for Delta lake support, however, you could write customized code using delta lake library to save output as a Delta lake.

To use Crawler to add meta of Delta lakes to Catalog, here is a workaround . The workaround is not pretty and has two major parts.

1) Get the manifest of referenced files of the Delta Lake. You could refer to Delta Lake source code, or play with the logs in _delta_log, or use a brutal method such as 

import org.apache.spark.sql.functions.input_file_name

spark.read.format("delta")
  .load(<path-to-delta-lake>)
  .select(input_file_name)
  .distinct

2) Use Scala or Python Glue API and the manifest to create or update table in Catalog.

user_62057430
Related