How to provide cluster name in Azure Databricks Notebook Run Now JSON

Viewed 278

I am able to use the below JSON through POSTMAN to run my Databricks notebook.

I want to be able to give a name to the cluster that is created through the "new_cluster" options.

Is there any such option available?

{
"tasks": [
    {
        "task_key": "Job_Run_Api",
        "description": "To see how the run and trigger api works",
        "new_cluster": {
            "spark_version": "9.0.x-scala2.12",
            "node_type_id": "Standard_E8as_v4",
            "num_workers": "1",
            "custom_tags": {
                "Workload": "Job Run Api"                    
            }
        },
        "libraries": [
            {
                "maven": {
                    "coordinates": "net.sourceforge.jtds:jtds:1.3.1"
                }
            }
        ],
        "notebook_task": {
            "notebook_path": "/Shared/POC/Job_Run_Api_POC",
            "base_parameters": {
                "name": "Junaid Khan"
            }
        },
        "timeout_seconds": 2100,
        "max_retries": 0            
    }
],
"job_clusters": null,
"run_name": "RUN_API_TEST",
"timeout_seconds": 2100
}

When the above API call is done, the cluster created has a name like "job-5975-run-2" and that is not super explanatory.

I have tried to use the tag "cluster_name" inside the "new_cluster" tag but I got an error that I can't do that, like this:

{
"error_code": "INVALID_PARAMETER_VALUE",
"message": "Cluster name should not be provided for jobs."
}

Appreciate any help here

1 Answers

Cluster name for jobs are automatically generated and can't be changed. If you want somehow track specific jobs, use tags.

P.S. If you want to have more "advanced" tracking capability, look onto Overwatch project.

Related