Apache Spark UI displays incorrect input size of file being ingested

user_836087 0
Viewed 834

My Java spark program ingests a file of 3.7 GB. When I launch the spark program and go to the Spark UI on port localhost:4040 The input size shown for the load stage is 7.3 GB??? That's really confusing. Why is the input size in the Spark UI console showing almost double than the actual file size being ingested?

enter image description here

apache-spark-sql apache-spark
1 Answers

The input size:

  • Is estimated.
  • Is not the input size of the file you load, but the input size of the loaded object, which in general, require more memory to store than a serialized objects (pointers to actual objects, overhead of the data structures used to load the data).
user_105536100
Related