I want manual control of proccess that devides data read to partitions. For example, suppose I have a DF
+---+---------------------+
|key|data |
+---+---------------------+
|k0 |[file0, file1, file2]|
|k1 |[file0, file3, file4]|
+---+---------------------+
And I have some col_i, col_j that I want to read from these files (in my case parquet) but I want to keep data in original partitions. So after udf call I expect to get something like
+---+------------------+------------------+
|key| col_i| col_j|
+---+------------------+------------------+
| k0|val_i_0_from_file0|val_j_0_from_file0|
| k0|val_i_1_from_file0|val_j_1_from_file0|
| k0|val_i_0_from_file1|val_j_0_from_file1|
| k0|val_i_0_from_file2|val_j_0_from_file2|
| k1|val_i_0_from_file0|val_j_0_from_file0|
| k1|val_i_1_from_file0|val_j_1_from_file0|
| k1|val_i_0_from_file3|val_j_0_from_file3|
| k1|val_i_0_from_file4|val_j_0_from_file4|
+---+------------------+------------------+
with all k0 rows on same partition (same for k1).
Why such a question: Table with these files is not partitioned by key column but has it in its structure. I want to join by this column. To avoid shuffle on this key after reading I manually read metainfo about these parquet files, filter them based on other join side (small dataframe) and repartition these links by key before reading.