How to partition a Delta Lake table by a range of dates?

user_19332088 0
Viewed 568

So I know that partitioning a table by date is pretty simple.

df.write.format("delta").mode("overwrite").partitionBy(day)

Now I want each partition to include all the data up to that date. So if the table has data from 2015, then I want a partition that includes data from 2015 to the current date (6/13/22). And tomorrow, I want the partition to have all the data from 2015 to the newest day (6/14/22).

Would I just do partitionBy(day WHERE DATE BETWEEN '2015-01-01' AND '2022-06-14')? OR does that just partition days between that date range? I'm a little confused with the SQL part.

delta-lake azure-databricks databricks apache-spark-sql
1 Answers

Try below code:

CREATE TABLE events
USING delta
PARTITIONED BY(date)
AS SELECT *, date(<date_column>) as date from <original_tablename>

where,

  • <Date_column> is the date column and,
  • <original_tablename> is the name of original table name.

Now if you want to merge the different partitioned tables, you can merge or join them based on the individual dates.

user_159690410
Related