Is there a way through which we can send our custom spark application logs to ELK ? I have checked this https://community.databricks.com/s/question/0D53f00001GHVj6CAH/how-to-push-cluster-logs-to-elastic-search
But I couldn't find a way through which we can specify the credentials to send the logs to ELK ?
Also, I have enabled Diagnostic settings on my Azure databricks workspace to write to Elastic search as partner solution. But its picking and writing only Workspace related logs and not custom application logs. https://docs.microsoft.com/en-us/azure/partner-solutions/elastic/manage