I'm trying to get the time an application spends in the state RUNNING. This can be easily done if a SparkLauncher is used by adding SparkAppHandle.Listener callbacks and listen to the stateChanged event.
Long story short, I don't have the option to use SparkLauncher but I need to know the actual runtime of an application (without the time it spends in the WAITING state).
Can this information be gleaned from the SparkContext? SparkListener doesn't seem to provide a callback for application state change. The actual duration cannot be worked out from onApplicationStart and onApplicationEnd because onApplicationStart fires when the application starts waiting.
I'm also aware of QueryExecutionListener but the individual jobs also include the WAITING period (onSuccess callback duration param).
Edit: Upon further inspection, SparkAppHandle.Listener doesn't seem to be able to do this either.