"java.lang.IllegalArgumentException: No valid credential configuration discovered" error when authenticating Hadoop-GCS connector

user_15719448 0
Viewed 1076

I am trying to load data into GCP using Spark from outside of GCP (from one of our on-prem clusters). To do that, I wrote the following code.

val conf = new SparkConf()
  .set("spark.network.timeout", "12000s")
  .set("spark.kryoSerializer.buffer.max", "512m")
  .set("temporaryGcsBucket", "GCS_BUCKET_LOCATION")
  .setAppName(args(1))
conf.registerKryoClasses(Array(classOf[Loader]))

val spark = SparkSession.builder().
  appName("app").
  master("yarn").
  config( "spark.serializer", "org.apache.spark.serializer.KryoSerializer").
  config(conf).
  getOrCreate()
spark.conf.set("temporaryGcsBucket", "GCS_BUCKET_LOCATION")
spark.conf.set("parentProject", "PROJECTID")

spark.sparkContext.hadoopConfiguration.set("google.cloud.auth.service.account.enable", "true")
spark.sparkContext.hadoopConfiguration.set("fs.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem")
spark.sparkContext.hadoopConfiguration.set("google.cloud.auth.service.account.json.keyfile", "/path/to/application_default_credentials.json")
spark.sparkContext.hadoopConfiguration.set("fs.gs.auth.client.id", "CLIENTID")
spark.sparkContext.hadoopConfiguration.set("fs.gs.auth.client.secret", "SECRETKEY")
spark.sparkContext.hadoopConfiguration.set("fs.AbstractFileSystem.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS")
spark.sparkContext.hadoopConfiguration.set("fs.gs.project.id", "PROJECTID")

Spark Read:

val dataframe = spark.read.format("jdbc")
          .option("url", s"jdbc:sqlserver://$server;DatabaseName=$databasename")
          .option("driver", "com.microsoft.sqlserver.jdbc.SQLServerDriver")
          .option("user", "USERNAME")
          .option("password", "PASSWORD")
          .option("dbtable", query)
          .option("partitionColumn", partitionColumn)
          .option("lowerBound", SOMEVALUE)
          .option("upperBound", SOMEVALUE)
          .option("numPartitions", numPartitions)
          .option("fetchsize", 10000)
          .load()

Spark Write:

dataframe.drop(s"${partitionColumn}1")
          .write.format("parquet")
          .option("parentProject", "PROJECTNAME")
          .option("path", s"gs://SOME_GCS_PATH/${tablename}")
          .option("credentialsFile","/path/of/jsonfile/on/local/application_default_credentials.json")
          .mode("append")
          .save()

When I run the code, I see the below error message:

java.lang.IllegalArgumentException: No valid credential configuration discovered:  [CredentialOptions{serviceAccountEnabled=false, serviceAccountPrivateKeyId=<redacted>, serviceAccountPrivateKey=<redacted>, serviceAccountEmail=hadoop-distp-sa-hadoop@ktest.imp.gserviceaccount.com, serviceAccountKeyFile=null, serviceAccountJsonKeyFile=/path/of/jsonfile/on/local/application_default_credentials.json, nullCredentialEnabled=false, transportType=JAVA_NET, tokenServerUrl=https://oauth2.googleapis.com/token, proxyAddress=null, proxyUsername=null, proxyPassword=null}]

So I tried to add the configuration as per the method3 present in Authentication section of this page.

and made the following changes to my configuration:

spark.sparkContext.hadoopConfiguration.set("google.cloud.auth.service.account.enable", "false")
spark.sparkContext.hadoopConfiguration.set("fs.gs.auth.impersonation.service.account.for.user.<USER_NAME>", "username@companyname.com")
spark.sparkContext.hadoopConfiguration.set("fs.gs.auth.impersonation.service.account.for.group.<GROUP_NAME>", "gcp_projectid")
spark.sparkContext.hadoopConfiguration.set("fs.gs.auth.impersonation.service.account", "service-account-name@.iam.gserviceaccount.com")

But I am still running into the same exception even after changing the configuration as per the above mentioned link.

Is there any thing that I am doing wrong here ? Could anyone let me know how can I fix the issue and what are the mandatory configuration parameters to provide in spark code to authenticate with GCP ? Any help is really appreciated.

google-cloud-storage pyspark apache-spark google-cloud-platform
2 Answers

As I understand from the documentation you've linked, you're using the GCS connector, therefore I believe that:

spark.sparkContext.hadoopConfiguration.set("fs.gs.auth.impersonation.service.account.for.user.<USER_NAME>", "username@companyname.com")
spark.sparkContext.hadoopConfiguration.set("fs.gs.auth.impersonation.service.account.for.group.<GROUP_NAME>", "gcp_projectid")

Should actually contain the same values as:

spark.sparkContext.hadoopConfiguration.set("fs.gs.auth.impersonation.service.account", "service-account-name@.iam.gserviceaccount.com")

Because the <USER_NAME> and <GROUP_NAME> sections ought to be changed to username@companyname.com and gcp_projectid respectively, at least according to the section in the documentation pertaining to these values
Given that the value for the third configuration is the account, I believe the others should be too... They're just there in case you want to specify user-specific or group-specific service accounts to use.

That being said, I believe those values are being ignored, as there's likely no user with the username `<USER_NAME> in your project, and instead would like to know a little more about this configuration:

spark.sparkContext.hadoopConfiguration.set("google.cloud.auth.service.account.enable", "false")

As you mention it as being part of the previous documentation, but I cannot seem to find it there? The only instance I can quickly find of said configuration option has it enabled, which is to say that the option is set to "true", without this option a service account won't be used, which seems odd considering you have declared which service account to use. Or at least that's the information I could glean from the following descriptive text:

    Whether to use a service account for GCS authorization.
    Setting this property to `false` will disable use of service accounts for
    authentication.

However, if that's the case, then it should most definitely warn you when this property is set to false and the other is set to a value that is not null, I believe.

So, reading over your post again... I believe you're overwriting the original configuration you've put with the "following changes to the configuration", but those changes do not include google.cloud.auth.service.account.json.keyfile, which I believe is necessary, but I can't say for sure, as the only things I've found within Google's docs are using Cloud Interconnect, so it's hard for me to say whether or not it's applicable in your case, as you're doing this from Spark and are not using DiskCTP.

user_154525720

The error is due to an incompatible argument used in the authentication configuration. The below authentication configurations have been removed from the Hadoop-GCS connector since January, 2021 and are not supported in the latest version. Connector version 2.1.6 was the last to support these configuration parameters. You can refer to the change history here (point #24).

fs.gs.auth.client.id
fs.gs.auth.client.file
fs.gs.auth.client.secret

To authenticate with a service account JSON key file, use the following configuration.

spark.sparkContext.hadoopConfiguration.set("fs.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem")
spark.sparkContext.hadoopConfiguration.set("fs.AbstractFileSystem.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS")
spark.sparkContext.hadoopConfiguration.set("google.cloud.auth.service.account.enable", "true")
spark.sparkContext.hadoopConfiguration.set("google.cloud.auth.service.account.json.keyfile", "/path/to/service_account_json_key_file.json")

The supported authentication methods in the latest version (v2.2.3) of the connector are discussed in the connector configuration properties. Quoting required parts from the md file.

Service account authentication

The following properties are required only when running not on a GCE VM and fs.gs.auth.service.account.enable is true. There are 3 ways to configure these credentials, which are mutually exclusive.

Method 1: Configure service account details directly in the Configuration file or via Hadoop Credentials.

Method 2: Configure service account credentials using a json keyfile.

Method 3: Configure service account credentials using a P12 certificate.

Service account impersonation

Service account impersonation can be configured for a specific user name and a group name, or for all users by default using below properties:

  • fs.gs.auth.impersonation.service.account.for.user.<USER_NAME> (not set by default)

The service account impersonation for a specific user.

  • fs.gs.auth.impersonation.service.account.for.group.<GROUP_NAME> (not set by default)

The service account impersonation for a specific group.

  • fs.gs.auth.impersonation.service.account (not set by default)

Default service account impersonation for all users.

If any of the above properties are set then the service account specified will be impersonated by generating a short-lived credential when accessing Google Cloud Storage.

Configured authentication method will be used to authenticate the request to generate this short-lived credential.

user_157458840
Related