With Delta Lake, how to remove original file after compaction

user_16341337 0
Viewed 577

Basically I have a spark streaming job (with delta) writing a small file to hdfs every 5min . I also have a compaction job that runs everyday to compact data from previous day into some big files (# of files depend on job repartition number). The big files are in the same directory as the original small files. Is there any way to effectively remove the original small files as they are useless?

I have tried the vacuum function for delta tables, but that basically removes all data out of retention period, regardless of compaction or not.

Here's how I compact my data (I'm using Java):

spark.read()
        .format("delta")
        .load(path) // hdfs path of the data
        .where(whereCondition) // my data is partitioned by date, so here should be "date = '2021-06-29'"
        .repartition(repartitionNum)
        .write()
        .option("dataChange", "false")
        .format("delta")
        .mode("overwrite")
        .option("replaceWhere", whereCondition)
        .save(path);

It would be great if anyone can tell me

  • If I'm doing it correctly with compaction
  • How to remove original small files which shouldn't be referenced by delta anymore.

Any comment is appreciated, thanks!

delta-lake spark-streaming databricks apache-spark
1 Answers

  1. a) You may also consider using coalesce instead of repartition. Because coalesce is more efficient than repartition. However, coalesce can be used only for decreasing the no. of files but repartition can be used for either decreasing or increasing no. of files. But in compaction, we need to decrease the no. of files always. So, I believe coalesce will be better than repartition. b) If you are using Databricks, then you may consider using OPTIMIZE command for compaction.

  2. For deleting old files, you need to use vacuum. There is no other way to do that.

user_156602350
Related