Terraform- GCP Data Proc Component Gateway Enable Issue

user_16175883 0
Viewed 100

I’m trying to create data proc cluster in GCP using terraform resource google_dataproc_cluster. I would like to create Component gateway along with that. Upon seeing the documentation, it has been stated as to use the below snippet for creation:

cluster_config {
 endpoint_config {
 enable_http_port_access = "true"
 }
}

Upon running the terraform plan, i see the error as " Error: Unsupported block type". And also tried using the override_properties and in the GCP data proc, i could see that the property is enabled, but still the Gateway Component is disabled. Wanted to understand, is there an issue upon calling the one given in the Terraform documentation and also is there an alternate for me to use it what?

software_config {
image_version = "${var.image_version}"
override_properties = {
"dataproc:dataproc.allow.zero.workers" = "true"
"dataproc:dataproc.enable_component_gateway" = "true"
}
}

The below is the error while running the terraform apply.

Error: Unsupported block type
 
 on main.tf line 35, in resource "google_dataproc_cluster" "dataproc_cluster":
35:           endpoint_config {
 Blocks of type "endpoint_config" are not expected here.

RESOURCE BLOCK:

resource "google_dataproc_cluster" "dataproc_cluster" {
  name     = "${var.cluster_name}"
  region   = "${var.region}"
  graceful_decommission_timeout = "120s"
  labels =  "${var.labels}"
  

  cluster_config {
 
        staging_bucket        = "${var.staging_bucket}"
        /*endpoint_config {
               enable_http_port_access = "true"
        }*/
       
        software_config {
            image_version       = "${var.image_version}"
            override_properties = {
                "dataproc:dataproc.allow.zero.workers" = "true"
                "dataproc:dataproc.enable_component_gateway" = "true"  /* Has Been Added as part of Component Gateway Enabled which is already enabled in the endpoint_config*/
            }
        }
         gce_cluster_config {
         // network          = "${var.network}"
          subnetwork       = "${var.subnetwork}"
          zone                = "${var.zone}"
         //internal_ip_only    = true
          tags                = "${var.network_tags}"
          service_account_scopes = [
              "cloud-platform"
      ]
        }

        master_config {
            num_instances     = "${var.master_num_instances}"
            machine_type      = "${var.master_machine_type}"
            disk_config {
                boot_disk_type    = "${var.master_boot_disk_type}"
                boot_disk_size_gb = "${var.master_boot_disk_size_gb}"
                num_local_ssds    = "${var.master_num_local_ssds}"
            }
        }

       
  }
  
  depends_on = [google_storage_bucket.dataproc_cluster_storage_bucket]

  timeouts {
    create = "30m"
    delete = "30m"
  }
}
terraform-provider-gcp terraform google-cloud-dataproc
2 Answers

This issue is discussed in this Git thread.

You can enable the component gateways in Cloud Dataproc by using google-beta provider in the Dataproc cluster and root configuration of terraform.

sample configuration:

  # Terraform configuration goes here

   provider "google-beta" {
   project = "my-project"
   
   }

  resource "google_dataproc_cluster" "mycluster" {
  provider = "google-beta"
  name     = "mycluster"
  region   = "us-central1"
  graceful_decommission_timeout = "120s"
  labels = {
    foo = "bar"
  } 
  ...
  ...
  }
user_157504730

Below is the snippet that worked for me to enable component gateway in GCP

provider "google-beta" {
  project     = "project_id"
}

resource "google_dataproc_cluster" "dataproc_cluster" {
  name     = "clustername"
  provider = google-beta 
  region   = us-east1
  graceful_decommission_timeout = "120s"
  

  cluster_config {
        endpoint_config {
               enable_http_port_access = "true"
        }
}
user_161758830
Related