What's significant change in spark 3.1.2 from 3.0.2 that causes Disk Space Used while enough Memory is available

user_16288561 0
Viewed 136

As it's seen in this snippet for Spark 3.1.2 is only running for 3+ hours and enough Memory is available but still it's using Disk Space Spark 3.1.2 UI Image

And this image shows for Spark 3.0.2 for Spark Streaming Job running for more than 90 hours no Disk Space is used Spark 3.0.2 UI Image

In both code is same it's just Spark version that's got updated. I have also noticed that Spark 3.1.2 is using more Memory than Spark 3.0.2. Is there anything changed in GC Collection as well? Any help on this is highly appreciated.

apache-spark
1 Answers

I am running into similar issues when upgrading from 3.1.1. I was able to compare the default settings across both versions using the logs that are printed after a cluster starts.

I noticed the Global Flag for AllocatePrefetchInstr is different in 3.1.2 but I haven’t been able to update the value to test performance.

Look into the default settings printed in the log.

user_90700010
Related