Cannot parametrize any value under placement.managedCluster.config

Viewed 141

My goal is to create dataproc workflow template from python code. Meanwhile I want to have ability to parametrize placement.managedCluster.config.gceClusterConfig.subnetworkUri field during template instantiation.

I read template from json file like:

{
  "id": "bigquery-extractor",
  "placement": {
    "managed_cluster": {
      "config": {
        "gce_cluster_config": {
          "subnetwork_uri": "some-subnet-name"
        },
        "software_config" : {
          "image_version": "1.5"
        }
      },
      "cluster_name": "some-name"
    }
  },
  "jobs": [
    {
      "pyspark_job": {
        "args": [
          "job_argument"
        ],
        "main_python_file_uri": "gs:///path-to-file"
      },
      "step_id": "extract"
    }
  ],
  "parameters": [
    {
      "name": "CLUSTER_NAME",
      "fields": [
        "placement.managedCluster.clusterName"
      ]
    },
    {
      "name": "SUBNETWORK_URI",
      "fields": [
        "placement.managedCluster.config.gceClusterConfig.subnetworkUri"
      ]
    },
    {
      "name": "MAIN_PY_FILE",
      "fields": [
        "jobs['extract'].pysparkJob.mainPythonFileUri"
      ]
    },
    {
      "name": "JOB_ARGUMENT",
      "fields": [
        "jobs['extract'].pysparkJob.args[0]"
      ]
    }
  ]
}

code snippet I use:

        options = ClientOptions(api_endpoint="{}-dataproc.googleapis.com:443".format(region))
        client = dataproc.WorkflowTemplateServiceClient(client_options=options)
        template_file = open(path_to_file, "r")
        template_dict = eval(template_file.read())
        print(template_dict)

        template = dataproc.WorkflowTemplate(template_dict)

        full_region_id = "projects/{project_id}/regions/{region}".format(project_id=project_id, region=region)
        try:
            client.create_workflow_template(
                parent=full_region_id,
                template=template
            )
        except AlreadyExists as err:
            print(err)
            pass

when I try to run this code I get the following error:

google.api_core.exceptions.InvalidArgument: 400 Invalid field path placement.managed_cluster.configuration.gce_cluster_config.subnetwork_uri: Field gce_cluster_config does not exist.

This behavior is the same also if I try to parametrize placement.managedCluster.config.softwareConfig.imageVersion, I will get

google.api_core.exceptions.InvalidArgument: 400 Invalid field path placement.managed_cluster.configuration.software_config.image_version: Field software_config does not exist.

But if I exclude any field under placement.managedCluster.config from parameters map, template is created successfully.

I didn't find any restriction on parametrizing these fields. Is there any? Or is it just me doing something wrong?

1 Answers

This doc listed the parameterizable fields. It seems that only managedCluster.name of managedCluster is parameterizable:

Managed cluster name. Dataproc will use the user-supplied name as the name prefix, and append random characters to create a unique cluster name. The cluster is deleted at the end of the workflow.

I don't see managedCluster.config parameterizable.

Related