we are processing data stored in a hive table using spark SQL (v2.1.0). we find that our spark tasks exhibit a heavy skew. e.g.: in below p_75 of the tasks had an input size of 127.1MB but the max input size was 634.7MB:
How can we get a more even distribution of the input size over the tasks?
we tried changing the # of partitions while reading data but it does not fix the skew:
Below are some other helpful properties of our hive table:
| ROW FORMAT SERDE |
| 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' |
| STORED AS INPUTFORMAT |
| 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat' |
| OUTPUTFORMAT |
| 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'
The HDFS files themselves are stored in parquet format with snappy compression (files end in .snappy.parquet) which is splittable per this link