Partition data by multiple partition keys - Azure ADF

Viewed 38

I have some data on on-prem SQL table. The data is huge ~100GB. The data many columns but two important ones are d_type and d_date.

d_type unique elements are 1,10,100 and d_date ranges from 2022-01-01 - 2022-03-30

I want to load this data into Azure using copy activity or dataflow but in a partitioned fashion, like the following format:

someDir/d_type=1/
                2022-01/somedata.parquet
                2022-02/somedata.parquet
                2022-03/somedata.parquet

someDir/d_type=10/
                2022-01/somedata.parquet
                2022-02/somedata.parquet
                2022-03/somedata.parquet


someDir/d_type=100/
                2022-01/somedata.parquet
                2022-02/somedata.parquet
                2022-03/somedata.parquet

I have tried with copy activity:

  1. Copy activity can only use one partition key
  2. If I partition by d_type, it creates parquet file with random bins i.e 1-20 (which contains only data for d_type=1), other file could have bins be 20-30 (which has no data)

Dataflow allows multiple partition keys, but I cannot use that sinceill have to copy the entrire data first from onprem sql to azure then process it. (As dataflow can only work with source link service which are linke via AzureIR and not SHIR).

Anyone got tips on how to solve this?

0 Answers
Related