How to read some partitions of delta table?

user_16034298 0
Viewed 452

I have a partitioned delta table stored in ADLS (partitoned on date column).

How to read only that data which is of the past one year, i.e data is something 2020-**-**?

delta-lake azure-databricks databricks
2 Answers

Load in partitioned raw table based on date

raw_df = spark.read.format('delta').load(raw_data_path + 'tablename/partitionname=specifieddate')

This is the code I use usually to retrieve a certain partition from ADLS

user_197942870

You just need to read table and filter out the data you need - Spark will perform predicate pushdown & will read only data from the matching partitions. It's simple as (assuming that column is called date):

df = spark.read.format("delta").load("your-table-root-path") \
  .filter("date >= '2020-01-01' and date <= '2020-12-31'")
Alex_Ott_186270
Related