Top Answers

1. Partitioning a large skewed dataset in S3 with Spark's partitionBy method
   22 votes
2. Why does the repartition() method increase file size on disk?
   20 votes
3. Limit kafka batch size when using Spark Structured Streaming
   10 votes
4. How does Parquet handles SparseVector Columns?
   6 votes
5. Why doesn't dataset's foreach method require an encoder, but map does?
   5 votes