Do Dataset Projections benefit from Data Set partitioning

twinkle2 0
Viewed 61

We would like to use Projections to speed up filtering and joins on a large incremental dataset with thousands of small (kb size) files.

Is it recommended to partition ( transforms.api.IncrementalTransformOutput.write_dataframe() with partitionBy=[col1, col2,...]) the main data set, to reduce the number of files, or would this be redundant effort, because it is done by Projections anyway?

If it is recommended to optimize the main data set, are there guidelines as to when this should be done?

palantir-foundry
1 Answers

Great question! As Spark makes use of a distributed file store, there are many techniques you can use to improve performance. The answer lies in what you want to do with the data after you repartition. The performance needs to be empirically tested (try a setting and see if it improves), and look at the build report to see the spark details. There are a couple of techniques you can use:

  1. Basic techniques: Use a single partition and do things in memory for small datasets: If your data is very small, you can use techniques such as a broadcast join to keep it memory. You should also repartition your data to a reasonable size to trade-off between overhead and parallelization. The rule of thumb, I use is keep partitions between 100-500 MB. You should also filter out any data you don’t need or drop entire columns. Spark isn’t very smart so keep filters simple and clear for optimal performance. Finally, make sure your data is clean and consistent. For example, modify 'Spark' and 'spark' to use consistent case and spacing. Otherwise they would have two different codes and Spark will read it as two different values.

  2. Hash Partitioning: For when downstream computations will need to match row keys (aggregates, joins, etc), and/or when pre-sorting will also help speed up different use cases. If you are doing many joins, you should use hash partitioning, and don't forget to repartition before saving the data.

    df = df.repartition(200) output.write_dataframe(df,bucket_cols=["patient_id"],bucket_count=200,sort_by=["patient_id"])

  3. Hive Partitioning: Large datasets where you want to have large amounts of pruning during filtering and have low cardinality columns. Only do this if you are doing a lot of filtering on a dataset with low cardinality. If you Hive partition on a dataset with high cardinality, you’ll end up with too many small files.

    output.write_dataframe(df, partition_cols=["date"])

I recommend you write a specific question with a minimal verifiable example and I can provide a more in-depth, specific answer for your use case.

Andrew Andrade0
Related