Google Cloud Dataproc cluster creation failure due to Cloud SQL proxy initialization error

Viewed 82

I am trying to create a Dataproc cluster from a Cloud Composer DAG using the DataprocCreateClusterOperator of Airflow. I need to access Cloud SQL from my Dataproc cluster hence need to install the Cloud SQL proxy on the cluster as well. I am providing the initialization action for Cloud SQL Proxy as per docs in my cluster config for the Airflow operator as below:

"initialization_actions": [
        {
           "executable_file" : "gs://<<some_gcs_bucket>>/cloud-sql-proxy.sh"
        }
    ],

"gce_cluster_config": {
        "service_account_scopes": ["https://www.googleapis.com/auth/cloud-platform","https://www.googleapis.com/auth/sqlservice.admin"],
        "metadata" : {
             "enable-cloud-sql-hive-metastore" : "false",
             "additional-cloud-sql-instances" : "<<PROJECT_ID>>:<<REGION>>:<<INSTANCE_NAME>>"
        }
    }

The cluster creation fails with below message:

google.api_core.exceptions.InvalidArgument: 400 Initialization action failed. Failed action 'gs://<<some_gcs_bucket>>/cloud-sql-proxy.sh', see output in: gs://<<some_gcs_bucket>>/dataproc-initialization-script-0_output

I might be putting the parameters in the cluster config wrongly like the metadata arguments required by the initialization script, if anybody has implemented this use-case previously then would appreciate your inputs.

1 Answers

The problem was specific to CentOS and Rocky. The support for CentOS and Rocky has been added in this PR. To use it, you can copy the source code from Github and upload it to your own GCS bucket.

Related