Data Factory / Data Flow - conditional split based on a number of records

user_6287629 0
Viewed 1059

I need to split a huge dataset into multiple files and each file must not have more than 100 000 rows. I don't know if this is possible with Data Flow and the conditional split?

azure-data-flow azure-data-factory
2 Answers

If you want simply split by a fixed number of rows, I've created a simple test.

  1. Declare a parameter inside the dataflow to store the row count of your source dataset. If your source dataset is Azure sql, you can use Lookup activity to get the max Row_No. If your source dataset is Azure storage, you can use Azure Function activity to get the max Row_No. Then pass the value to the parameter. enter image description here

  2. Here for test, set a static default value. enter image description here

  3. Then we can set Number of partitions expression $RowCount/10, if you want 10 lines per file. enter image description here

  4. We can set file names after division here.
    enter image description here

  5. My source dataset contains 50 lines, so ADF will split it to 5 files. Judging by the Id column, it has randomly taken 10 rows of data. enter image description here

user_139794870

You can achieve this with 2 dataflows. 1 to get the row count and another to partition. This can also be achieved in 1 dataflow using a cache sink in the future.

user_147921360
Related