I'm implementing custom spark listener in Java via:
public class MetricsSparkListener extends SparkListener
and want to capture Executor metrics as displayed in Spark history UI:
Spark UI Executor Tab
Can someone please share some sample code or API documentation how to do this. I can't find any example on the web or any description.
I'm trying onExecutorMetricsUpdate(SparkListenerExecutorMetricsUpdate executorMetricsUpdate) but can't really get to contents of ExecutorMetrics object.
Call to executorMetricsUpdate.executorUpdates().get().getMetricValue(ExecutorMetricType metricType) requires ExecutorMetricType parameter which seems to be some Scala interface. Not sure what how to create metric type value.
Eventually I would like to reach to executor metrics described here Available Executor Metrics especially peak memory metrics and have them summarized for the whole app.