Why is Size in Memory >> Size on disk for an RDD in spark?

Viewed 15

For a particular RDD, when I check out the spark UI, the Size in memory is 143.8GiB whereas the Size on disk is 4.7Gib. Why is the size in memory so large? Is it because the data stored on disk is in a highly compressed format(parquet)?

0 Answers
Related