Google Cloud Storage requires storage.objects.create permission when reading from pyspark

user_2633360 0
Viewed 1682

I'm trying to read pyspark DataFrame from Google Cloud Storage, but I keep getting an error that the service account has no storage.objects.create permissions. The account does not have WRITER permissions, but it's just reading parquet files:

spark_session.read.parquet(input_path)

18/12/25 13:12:00 INFO com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl: Repairing batch of 1 missing directories.
18/12/25 13:12:01 ERROR com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl: Failed to repair some missing directories.
com.google.cloud.hadoop.repackaged.gcs.com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
{
  "code" : 403,
  "errors" : [ {
    "domain" : "global",
    "message" : "***.gserviceaccount.com does not have storage.objects.create access to ***.",
    "reason" : "forbidden"
  } ],
  "message" : "***.gserviceaccount.com does not have storage.objects.create access to ***."
}
airflow google-cloud-storage pyspark apache-spark-sql google-cloud-platform
2 Answers

We found the issue. It's due to the implicit auto repair feature in the GCS connector. We disabled this behavior by setting fs.gs.implicit.dir.repair.enable to false.

user_26333600

Please see this question: Why does Spark running in Google Dataproc store temporary files on external storage (GCS) instead of local disk or HDFS while using saveAsTextFile?

Spark will create temporary files when doing some actions. I have run into this when extracting data from GCS files and converting to a user-defined object. It can also do this when loading into BQ because writing to Storage and then executing a single load from GCS is more efficient. You can see the change which did that here.

Sadly, there is no concrete link I can give you because the problem is not documented, as far as I know. I will try to find one for you and will update my response if I succeed.

user_42972100
Related