Access Denied when performing S3 Multipart Upload operation into SSE-KMS encrypted bucket

user_7281681 0
Viewed 256

I'm facing Access Denied for Multipart Upload into SSE-KMS encrypted bucket.
Code is running Glue (might be from other services, have no ability to verify). I’ve tried set of different permissions, even full access, no effect.

  1. Access for KMS is granted properly (kms:Decrypt, kms:Encrypt and kms:GenerateDataKey*) and it worked previously!
  2. This issues appears for both Glue on Spark and Glue PyShell jobs and do not affect small files.
  3. For Glue PySpark jobs this issue appear only when Glue security configuration is set (the key is same as granted for the job)
  4. For PySpark, writing is done via 
    output_sink = glueContext.getSink(...)
output_sink.writeFrame(dynamic_frame)
  5. For PySpark, writing is done via 
    df = pandas.read_excel(...)
df.to_parquet(output_file_path, compression="snappy", index=False)
  6. There are no bucket policies applied at all except restricting public access

What I already tried with no success:

  1. Add s3:* permissions to resources
  2. Add kms:* permissions to the policy for KMS key used in the jobs
  3. Additionally add bucket ARN resources (without key prefixes)
  4. Setting s3:* + Resource: "*"
  5. Add glue.amazonaws.com to Principal Services for KMS key
  6. For PySpark job, specified manually fs.s3.enableServerSideEncryption and fs.s3.serverSideEncryption.kms.keyId with corresponding key ARN
  7. For PyShell job, tried upgrading awscli, botocore and boto3 versions (pandas=1.1.5 and s3fs=0.4.2 versions cannot be upgraded to higher versions due to PyShell python version of 3.6.13)

Part of the stacktrace for PySpark job indicating multipart upload issue:

: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1.0 (TID 4, 172.36.10.82, executor 1): com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: ..; S3 Extended Request ID: ..), S3 Extended Request ID: ....
..<cropped entries>..
    at com.amazon.ws.emr.hadoop.fs.s3.lite.executor.GlobalS3Executor.execute(GlobalS3Executor.java:110)
    at com.amazon.ws.emr.hadoop.fs.s3.lite.AmazonS3LiteClient.invoke(AmazonS3LiteClient.java:189)
    at com.amazon.ws.emr.hadoop.fs.s3.lite.AmazonS3LiteClient.invoke(AmazonS3LiteClient.java:184)
    at com.amazon.ws.emr.hadoop.fs.s3.lite.AmazonS3LiteClient.putObject(AmazonS3LiteClient.java:107)
    at com.amazon.ws.emr.hadoop.fs.s3n.Jets3tNativeFileSystemStore.storeFile(Jets3tNativeFileSystemStore.java:174)
    at com.amazon.ws.emr.hadoop.fs.s3n.MultipartUploadOutputStream.uploadSinglePart(MultipartUploadOutputStream.java:208)
    at com.amazon.ws.emr.hadoop.fs.s3n.MultipartUploadOutputStream.close(MultipartUploadOutputStream.java:423)
    at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:74)
    at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:108)
    at org.apache.parquet.nimble.hadoop.ParquetFileWriter.end(ParquetFileWriter.java:579)
..<cropped entries>..
    at com.amazonaws.services.glue.sinks.GlueParquetHadoopWriter.writeParquetPartitioned(GlueParquetHadoopWriter.scala:163)
    at com.amazonaws.services.glue.sinks.GlueParquetHadoopWriter$$anonfun$doParquetWrite$2.apply(GlueParquetHadoopWriter.scala:188)
    at com.amazonaws.services.glue.sinks.GlueParquetHadoopWriter$$anonfun$doParquetWrite$2.apply(GlueParquetHadoopWriter.scala:181)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
    at org.apache.spark.scheduler.Task.run(Task.scala:121)
..<cropped entries>..

Error message for PyShell:

Sending http request: <AWSPreparedRequest stream_output=False, method=PUT, 
url=https://my-bucket-name.s3.ca-central-1.amazonaws.com/folder/folder/folder/file-name.snappy.parquet?partNumber=1&uploadId=~uploadId~, 
headers={
  'User-Agent': b'Botocore/1.12.232 Python/3.6.13 Linux/4.14.238-125.422.amzn1.x86_64',
  'Content-MD5': b'Ic4VG7BgETssQJOhSK+E/Q==',
  'Expect': b'100-continue',
  'X-Amz-Date': b'20220518T163248Z',
  'X-Amz-Security-Token': b'~token-data~',
  'X-Amz-Content-SHA256': b'UNSIGNED-PAYLOAD',
  'Authorization': b'AWS4-HMAC-SHA256 Credential=~credential~, SignedHeaders=content-md5;host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=~signature~',
  'Content-Length': '5421349'
}>
...

Traceback (most recent call last):
  File "/tmp/glue-python-scripts-2tscdixy/script.py", line 44, in main
    df.to_parquet(output_file_path, compression="snappy", index=False)
  File "/glue/lib/installation/pandas/util/_decorators.py", line 199, in wrapper
    return func(*args, **kwargs)
  File "/glue/lib/installation/pandas/core/frame.py", line 2372, in to_parquet
    **kwargs,
  File "/glue/lib/installation/pandas/io/parquet.py", line 276, in to_parquet
    **kwargs,
  File "/glue/lib/installation/pandas/io/parquet.py", line 123, in write
    self.api.parquet.write_table(table, path, compression=compression, **kwargs)
  File "/glue/lib/installation/pyarrow/parquet.py", line 2034, in write_table
    writer.write_table(table, row_group_size=row_group_size)
  File "/glue/lib/installation/pyarrow/parquet.py", line 686, in __exit__
    self.close()
  File "/glue/lib/installation/pyarrow/parquet.py", line 710, in close
    self.file_handle.close()
  File "pyarrow/io.pxi", line 173, in pyarrow.lib.NativeFile.close
  File "/glue/lib/installation/fsspec/spec.py", line 1630, in close
    self.flush(force=True)
  File "/glue/lib/installation/fsspec/spec.py", line 1501, in flush
    if self._upload_chunk(final=force) is not False:
  File "/glue/lib/installation/s3fs/core.py", line 1245, in _upload_chunk
    raise IOError('Write failed: %r' % exc)
OSError: Write failed: ClientError('An error occurred (AccessDenied) when calling the UploadPart operation: Access Denied',)
amazon-kms aws-glue pyspark amazon-web-services amazon-s3
1 Answers

This issue persisted for a 3 days in a row and today it suddenly disappeared. My only suspicion that this issue was caused by AWS internal bug which has been fixed just yesterday and all non-working policies started to give proper permissions.

For someone who will be looking for any clue or solution, the only possible option I found useful (but it's not confirmed that this actually works) is to add separate policy statement with bucket list permission as well as add s3:ListMultipartUploadParts and s3:AbortMultipartUpload actions for the target folder statement:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:ListBucketMultipartUploads",
        "s3:GetBucketLocation"
      ],
      "Resource": "arn:aws:s3:::my-bucket-name"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:GetEncryptionConfiguration",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:ListMultipartUploadParts",
        "s3:AbortMultipartUpload"
      ],
      "Resource": [
        "arn:aws:s3:::my-bucket-name/my-output-folder/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "kms:Encrypt",
        "kms:Decrypt",
        "kms:ReEncrypt*",
        "kms:GenerateDataKey*",
        "kms:DescribeKey"
      ],
      "Resource": "arn:aws:kms:region:11111:key/kkkkkk"
    }
  ]
}
user_72816810
Related