Spark input size when reading Parquet from S3 is 2x higher than from local FS

Viewed 88

I am doing some tests with Spark and Parquet. I have an identical Parquet file locally and on S3. I read the file with Spark with a simple query:

val data = sparkSession.read.parquet(path)
  .filter(col("mycol").startsWith("C")) // Partition filter
data.collectAsList() // just to force execution

When I point the path to S3, I have a much higher input size (+- 2x) than when I read from my local filesystem. The number of records is the same.

stats for local and with S3

Can anybody explain this?

First, I thought it had to do with the fact that S3 is block storage and that it reads an entire block. My partition file was smaller than 32MB so would be read two times (once for schema and once for values). But I have the same behavior with a Parquet file with one large partition (190MB)... I also tried out different queries and I have this result with any query I try.

Thank you for any help provided!

0 Answers
Related