Is there a way to retrieve the aws_attributes of the cluster running PySpark/Databricks programmatically?
Specifically, I'd like to get the information about which region the cluster is running so I can output to the same-region S3 bucket.
1 Answers
You can get it by calling Cluster Get REST API - it will return JSON including the aws_attributes. Something like this:
import requests
ctx = dbutils.notebook.entry_point.getDbutils().notebook().getContext()
host_name = ctx.tags().get("browserHostName").get()
host_token = ctx.apiToken().get()
cluster_id = ctx.tags().get("clusterId").get()
response = requests.get(
f'https://{host_name}/api/2.0/clusters/get?cluster_id={cluster_id}',
headers={'Authorization': f'Bearer {host_token}'}
).json()
aws_info = response['aws_attributes']
Related