I'm trying to get spark logs from Databricks and to check connection string that is sent to SNowflake. At what path, spark logs can be found?
I already have this tracing option enabled, but do i need to enable some additional logging for spark:
sfOptions = {
"sfURL" : "test.eu-central-1.snowflakecomputing.com",
"sfUser" : "user",
"sfPassword" : **********
"sfDatabase" : "db",
"sfSchema" : "sch",
"sfWarehouse" : "wh",
"tracing" : "all",
}
Thank you
It really depends on where the information will be logged - on drivers only, or on executors as well. If you navigate to Cluster UI, you'll see two options "Driver Logs" and "Spark UI". The first one will give you access to all driver logs for given cluster, and you can access executor & driver logs via second item - it's just a standard Spark UI options.
You can also configure delivery of the logs to DBFS, and in this case you can download them from specified location to local machine for more investigations (for example, via databricks-cli)