Generate signed S3 url using boto3 with specified storage class

user_2600531 0
Viewed 1822

I can successfully upload a file to S3 using both generate_presigned_url() and generate_presigned_post() without specifying a storage class.

As soon as I try to specify a storage class I receive an error.

import boto3
from botocore.client import Config

s3_client = boto3.client(service_name='s3',config=Config(signature_version='s3v4'))
s3_client.generate_presigned_url(ClientMethod='put_object',Params={'Bucket':bucket,'Key':'test.txt', 'StorageClass':'INTELLIGENT_TIERING'})

fails with

The request signature we calculated does not match the signature you provided. Check your key and signing method.

import boto3
from botocore.client import Config

s3_client = boto3.client(service_name='s3',config=Config(signature_version='s3v4')
parts=s3_client.generate_presigned_post(Bucket=bucket,Key='test.txt', Fields={'StorageClass':'INTELLIGENT_TIERING'})
files = {'file':open('test.txt')}
response=requests.post(parts['url'], data=parts['fields'], files=files)

fails with

Invalid according to Policy: Extra input fields: StorageClass

I've tried the latter approach with "Storage-Class", "x-amz-storage-class", and a variety of different key names with the same result.

Where am I going wrong?

UPDATE: more info

I don't believe I have any permission issues since this works:

s3_client.put_object(Bucket=bucket,Key='test.txt', StorageClass='INTELLIGENT_TIERING', Body=binary_data)

Also, I get the same signature mismatch error noted above with generate_presigned_url() if I replace INTELLIGENT_TIERING with STANDARD. Seems to not be an issue of the new class being unsupported.

boto boto3 amazon-web-services amazon-s3
2 Answers

Per this comment

By adding the 'StorageClass': 'STANDARD_IA' into the params you're including it as part of the signature as a signed header, this is just how S3 serializes the storage class. All headers that are signed need to be sent with the request when you used the presigned url. In this case you would need to include the following header in your PUT request:

headers = {'x-amz-storage-class': 'STANDARD_IA'}

I hadn't been including the header 'x-amz-storage-class' in my client when calling the url generated via generate_presigned_url because I'd assumed the url contained all the info S3 needed. Adding it resolved this.

user_26005310

More than likely the ability to define your storage class is being blocked by your AWS IAM permissions:

By default s3:PutObject stores objects using the STANDARD storage class, but you can use the x-amz-storage-class request header to specify a different storage class.

When granting the s3:PutObject permission, you can use the s3:x-amz-storage-class condition key to restrict which storage class to use when storing uploaded objects. For more information about storage classes, see Storage Classes.

https://docs.aws.amazon.com/AmazonS3/latest/dev/amazon-s3-policy-keys.html#example-storage-class-condition-key

Are you in a position to share the IAM permissions that is set for your S3 bucket upload?

Assuming you have access to the IAM permissions, you may find that a restriction was added as follows:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:PutObject",
            "Resource": [
                "arn:aws:s3:::examplebucket/*"
            ],
            "Condition": {
                "StringEquals": {
                    "s3:x-amz-storage-class": [
                        "STANDARD"
                    ]
                }
            }
        }
    ]
}
user_107837000
Related