I have several jobs scheduled in databricks, out of which I am interested in reading statistics of two specific jobs. I need to write a databricks notebook code to write the job statistics (jobName, startTime, endTime and status) into a snowflake table.
1 Answers
We can use following python code to get the details from databricks job api.
Note : tested code here
from pyspark.sql.types import IntegerType
from pyspark.sql.types import *
from pyspark.sql import Row
import base64
import requests
import json
databricks_instance ="<databricks-instances>"
url_list = f"{databricks_instance}/api/2.0/jobs/runs/get?run_id=39347"
headers = {
'Authorization': 'Bearer <databricks-access-token>',
'Content-Type': 'application/json'
}
response = requests.request("GET", url_list, headers=headers).json()
print(response)
print(response['job_id'])
print(response['start_time'])
print(response['end_time'])
Related