In stage of reading a text file of size 19GB, the Peak JVM memory goes till 26 GB if spark.executor.memory is configured as 100 GB whereas for the same file when we give just 1 GB executor memory, Peak JVM memory goes only till 600 MB.
There is no execution memory usage in this stage. So why is there difference in the peak jvm usage for two cases? which objects are occupying this extra memory in case of executor with 100 GB memory?
It's for Spark on K8s in cluster mode.