How to fix "NullPointerException: projectId must not be null" in Spark application on GKE?

user_1305344 0
Viewed 545

I'm deploying a Spark Structured Streaming application to Google Kubernetes Engine and while accessing a bucket using gs:// URI scheme I'm facing the following exception:

Exception in thread "main" java.lang.NullPointerException: projectId must not be null
    at com.google.cloud.hadoop.repackaged.gcs.com.google.common.base.Preconditions.checkNotNull(Preconditions.java:897)
    at com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl.createBucket(GoogleCloudStorageImpl.java:437)
    at com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.gcsio.GoogleCloudStorage.createBucket(GoogleCloudStorage.java:88)
    at com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.gcsio.GoogleCloudStorageFileSystem.mkdirsInternal(GoogleCloudStorageFileSystem.java:456)
    at com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.gcsio.GoogleCloudStorageFileSystem.mkdirs(GoogleCloudStorageFileSystem.java:444)
    at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.mkdirs(GoogleHadoopFileSystemBase.java:911)
    at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:2275)
    at org.apache.spark.sql.execution.streaming.StreamExecution.<init>(StreamExecution.scala:137)
    at org.apache.spark.sql.execution.streaming.MicroBatchExecution.<init>(MicroBatchExecution.scala:50)
    at org.apache.spark.sql.streaming.StreamingQueryManager.createQuery(StreamingQueryManager.scala:317)
    at org.apache.spark.sql.streaming.StreamingQueryManager.startQuery(StreamingQueryManager.scala:359)
    at org.apache.spark.sql.streaming.DataStreamWriter.startQuery(DataStreamWriter.scala:466)
    at org.apache.spark.sql.streaming.DataStreamWriter.startInternal(DataStreamWriter.scala:456)
    at org.apache.spark.sql.streaming.DataStreamWriter.start(DataStreamWriter.scala:301)
    at meetup.SparkStreamsApp$.delayedEndpoint$meetup$SparkStreamsApp$1(SparkStreamsApp.scala:25)
    at meetup.SparkStreamsApp$delayedInit$body.apply(SparkStreamsApp.scala:7)

I'm pretty sure it's related to a service account to access and create subdirectories in the bucket that I've been using while spark-submit the Spark app locally using GOOGLE_APPLICATION_CREDENTIALS environment variable and spark.hadoop.google.cloud.auth.service.account.enable=true configuration property.

I'm deploying the Spark application as follows:

./bin/spark-submit \
  --master k8s://$K8S_SERVER \
  --deploy-mode cluster \
  --name $POD_NAME \
  --class meetup.SparkStreamsApp \
  --conf spark.kubernetes.driver.request.cores=400m \
  --conf spark.kubernetes.executor.request.cores=100m \
  --conf spark.kubernetes.container.image=$SPARK_IMAGE \
  --conf spark.kubernetes.driver.pod.name=$POD_NAME \
  --conf spark.kubernetes.namespace=$K8S_NAMESPACE \
  --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
  --conf spark.kubernetes.submission.waitAppCompletion=false \
  --conf spark.hadoop.fs.gs.impl=com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem \
  --conf spark.hadoop.google.cloud.auth.service.account.enable=true \
  --verbose \
  local:///opt/spark/jars/meetup.spark-streams-demo-0.1.0.jar $BUCKET_NAME

How to fix it in a proper Kubernetes / GKE-way?

google-kubernetes-engine google-cloud-storage apache-spark kubernetes google-cloud-platform
2 Answers

Based on your configuration I would suggest you to add the following property fs.gs.project.id as indicated here. As it is showed up as Required. Google Cloud Project ID with access to configured GCS buckets.

Additional, to this I am agree with the statement from @blackbishop about the secret management.

user_108910370

The recommended way in GKE documentation is to Import credentials as a Secret :

kubectl create secret generic spark-streaming-sa --from-file=/path/spark-streaming-serviceaccount-key.json

And when you submit the job add the following configurations :

--conf spark.kubernetes.driver.secrets.spark-streaming-sa=<mount path>
--conf spark.kubernetes.executor.secrets.spark-streaming-sa=<mount path>
--conf spark.kubernetes.driverEnv.GOOGLE_APPLICATION_CREDENTIALS=<mount path>/spark-streaming-sa.json
--conf spark.executorEnv.GOOGLE_APPLICATION_CREDENTIALS=<mount path>/spark-streaming-sa.json
--conf spark.hadoop.google.cloud.auth.service.account.json.keyfile=<mount path>/spark-streaming-sa.json

You can refer to the examples provided on Github GoogleCloudPlatform/spark-on-k8s-gcp-examples.

This is also described in the Secret Management section of spark docs Running Spark on Kubernetes:

Kubernetes Secrets can be used to provide credentials for a Spark application to access secured services. To mount a user-specified secret into the driver container, users can use the configuration property of the form spark.kubernetes.driver.secrets.[SecretName]=<mount path>. Similarly, the configuration property of the form spark.kubernetes.executor.secrets.[SecretName]=<mount path> can be used to mount a user-specified secret into the executor containers.

user_13865510
Related