I am trying to create a service that looks for all the spark jobs' logs for a particular databricks workspace. Lets say I have deployed two spark streaming jobs in a databricks workspace - I would like to get the logs of those two jobs via any API or via message queues like kafka/eventhubs.
That way, I can subscribe to those streaming logs and take some events based actions in real time.
Is there any APIs available in Databricks workspace - which will give us the logs of each spark jobs or can we get those spark application's custom logs over an event hub ?