How to get memory used by Spark Executor

Question

I'm trying to understand if our memory configuration is ok or if we are over-allocating resources in our Executor, but I'm not able to figure out how to get the memory usage. Note that I've a limitation into getting information via Spark REST API.

What I can see if I extract the details for executor (using http://host:4072/api/v1/applications/app-20201109081013-5073/allexecutors) is (note that I've removed some data for clarity):

{
  "id" : "0",
  "hostPort" : "",
  "isActive" : true,
  "memoryUsed" : 50515998177,
  "diskUsed" : 0,
  "maxMemory" : 10119177830,
  "addTime" : "2020-11-09T08:10:15.933GMT",
  "memoryMetrics" : {
    "usedOnHeapStorageMemory" : 50515998177,
    "usedOffHeapStorageMemory" : 0,
    "totalOnHeapStorageMemory" : 10119177830,
    "totalOffHeapStorageMemory" : 0
  }

I can understand the maxMemory that is basically the allocated memory (10GB of 16GB due to memory region splitting)... but I'm puzzled about the 50GB that I could see as memoryUsed. This seems to be the sum of the storage memory used since the startup as it's always growing... Same information is shown into Executors tab into Spark UI.

So the question is: there's a way to get current used storage memory and, last but not least, the same for the execution memory?

Thank you very much

How to get memory used by Spark Executor

0 Answers