I am running a Spark streaming job where few millions of Kafka events are being processed in a sliding window of 3 minutes, with window size of 24 hrs. There are multiple steps in the job, including aggregating and filtering the events based on certain fields and joining them with static RDDs loaded from files in S3 and finally running an MLLib transformation on each aggregated row of the window, publishing the results to Kafka topic.
I need a way to publish a bunch of application metrics, starting with how much time it takes to complete processing for each window, how many raw events being processed, what's the data size in bytes being processed, etc.
I've searched through all events that Spark publishes, and executor level events doesn't get me what I need. I'm trying out Kamon and Spark MetricsSource and Sink for now.
Any suggestions on the best way to accomplish this?
Also, I'm using Spark 2.4 now, as the original codebase is pretty old. But will be migrating to Spark 3.x soon.