Databricks only provides cluster level logs in the UI or in the API. Is there a way to configure spark or log4j in databricks such that we get run/job level logs?
1 Answers
You can find a Guide on Monitoring Azure Databricks on the Azure Architecture Center, explaining the concepts used in this article - Monitoring And Logging In Azure Databricks With Azure Log Analytics And Grafana.
To provide full data collection, we combine the Spark monitoring library with a custom log4j.properties configuration. The build of the monitoring library for Spark 2.4 and the installation in Databricks is automated through the scripts referenced in the tutorial and available at https://github.com/algattik/databricks-monitoring-tutorial/.
Related