For a particular RDD, when I check out the spark UI, the Size in memory is 143.8GiB whereas the Size on disk is 4.7Gib. Why is the size in memory so large? Is it because the data stored on disk is in a highly compressed format(parquet)?
For a particular RDD, when I check out the spark UI, the Size in memory is 143.8GiB whereas the Size on disk is 4.7Gib. Why is the size in memory so large? Is it because the data stored on disk is in a highly compressed format(parquet)?