S3 presigned URL works 90 minutes after bucket creation

user_8991007 0
Viewed 865

We generate presigned URLs in order for users to upload files directly into S3 buckets. Running integration tests we discovered a failing test where an HTTP PUT request on a presigned URL yielded a SignatureDoesNotMatch error response. Surprisingly, the same code worked fine using another bucket. We kept trying on the original bucket that caused the test to fail and were surprised when it suddenly started to work without any real code changes.

We noticed that it was roughly two hours after we had created the bucket when the test successfully ran through. Since we are located at UTC+0200 we suspected the issue to be somehow related to that time difference and/or some clock synching issue. We set out to confirm our suspicions that the same presigned URL would suddenly just work after enough time has passed. SPOILER: It does!

The following code creates a brand new bucket, generates a presigned URL suitable for file upload (ClientMethod='put_object'), and tries to HTTP PUT some data using the requests library. We re-try PUTting data every 60 seconds until it finally succeeds 5419 seconds (or 90 minutes) after the bucket was created.

Note: Even though the bucket is deleted afterwards, running the same script (using the same bucket name) now instantly succeeds. In case you want to re-confirm this behavior, make sure to use a different bucket name the second time around.

import logging
import time

import boto3
import requests

from botocore.client import Config

logger = logging.getLogger(__name__)

# region = "eu-central-1"
# region = "eu-west-1"
# region = "us-west-1"
region = "us-east-1"
s3_client = boto3.client('s3', region_name=region, config=Config(signature_version='s3v4'))


if __name__ == "__main__":
    bucket_name = "some-globally-unique-bucket-name"

    key_for_file = "test-file.txt"

    # create bucket
    if region == "us-east-1":
        # https://github.com/boto/boto3/issues/125
        s3_client.create_bucket(Bucket=bucket_name, ACL='private')
    else:
        s3_client.create_bucket(Bucket=bucket_name, ACL='private',
                                CreateBucketConfiguration={'LocationConstraint': region})
    creation_time = time.time()

    # generate presigned URL
    file_data = b"Hello Test World"
    expires_in = 4 * 3600
    url = s3_client.generate_presigned_url(ClientMethod='put_object', ExpiresIn=expires_in,
                                           Params={'Bucket': bucket_name, 'Key': key_for_file})

    time_since_bucket_creation = time.time() - creation_time
    time_interval = 60
    max_time_passed = expires_in
    success = False
    try:
        while time_since_bucket_creation < max_time_passed:
            response = requests.put(url, data=file_data)
            if response.status_code == 200:
                success = True
                break

            if b"<Code>SignatureDoesNotMatch</Code>" in response.content:
                reason = "SignatureDoesNotMatch"
            else:
                reason = str(response.content)

            time_since_bucket_creation = time.time() - creation_time
            print("="*50)
            print(f"{time_since_bucket_creation:.2f} s after bucket creation")
            print(f"unable to PUT data to url: {url}")
            print(f"reason: {reason}")
            print(response.content)
            time.sleep(time_interval)
    except KeyboardInterrupt:
        print("Gracefully shutting down...")

    if success:
        print("YAY! File Upload was successful!")
        time_since_bucket_creation = time.time() - creation_time
        print(f"{time_since_bucket_creation:.2f} seconds after bucket creation")
        s3_client.delete_object(Bucket=bucket_name, Key=key_for_file)

    # delete bucket
    s3_client.delete_bucket(Bucket=bucket_name)

We run integration tests with an AWS EKS cluster where we create a cluster along with some databases, S3 buckets, etc. and tear everything down after the tests have completed. Having to wait 90 minutes for the presigning of URLs to work is not feasible.

My Questions
Am I doing anything wrong?
Is this expected behavior? Is there an acceptable workaround?
Can someone, please, confirm this behavior using the above code?

EDIT
I updated the code to create a bucket in the "us-east-1" region as suggested by "Michael - sqlbot" in the comments. The weird if statement is necessary as documented here. I am able to confirm Michael's suspicion that the behavior is NOT reproducible with "us-east-1".

In case it is of interest, the returned XML in the error case:

<?xml version="1.0" encoding="UTF-8"?>
<Error>
    <Code>SignatureDoesNotMatch</Code>
    <Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>
    <AWSAccessKeyId>REDACTED</AWSAccessKeyId>
    <StringToSign>AWS4-HMAC-SHA256
    20190609T170351Z
    20190609/eu-central-1/s3/aws4_request
    c143cb44fa45c56e52b04e61b777ae2206e0aaeed40dafc78e036878fa91dfd6</StringToSign>
    <SignatureProvided>REDACTED</SignatureProvided>
    <StringToSignBytes>REDACTED</StringToSignBytes>
    <CanonicalRequest>PUT
    /test-file.txt
    X-Amz-Algorithm=AWS4-HMAC-SHA256&amp;X-Amz-Credential=REDACTED%2F20190609%2Feu-central-1%2Fs3%2Faws4_request&amp;X-Amz-Date=20190609T170351Z&amp;X-Amz-Expires=14400&amp;X-Amz-SignedHeaders=host
    host:some-globally-unique-bucket-name.s3.eu-central-1.amazonaws.com

    host
    UNSIGNED-PAYLOAD</CanonicalRequest>
    <CanonicalRequestBytes>REDACTED</CanonicalRequestBytes>
    <RequestId>E6CBBC7D2E4D322E</RequestId>
    <HostId>j1dM1MNaXaDhzMUXKhqdHd6+/Rl1C3GzdL9YDq0CuP8brQZQV6vbyE9Z63HBHiBWSo+hb6zHKVs=</HostId>
</Error>
boto3 amazon-s3 python
1 Answers

Here's what you're bumping into:

A temporary redirect is a type of error response that signals to the requester that they should resend the request to a different endpoint. Due to the distributed nature of Amazon S3, requests can be temporarily routed to the wrong facility. This is most likely to occur immediately after buckets are created or deleted.

For example, if you create a new bucket and immediately make a request to the bucket, you might receive a temporary redirect, depending on the location constraint of the bucket. If you created the bucket in the US East (N. Virginia) AWS Region, you will not see the redirect because this is also the default Amazon S3 endpoint.

However, if the bucket is created in any other Region, any requests for the bucket go to the default endpoint while the bucket's DNS entry is propagated. The default endpoint redirects the request to the correct endpoint with an HTTP 302 response. Temporary redirects contain a URI to the correct facility, which you can use to immediately resend the request.

https://docs.aws.amazon.com/AmazonS3/latest/dev/Redirects.html

Note that the last part -- which you can use to immediately resend the request -- is not quite accurate. You can -- but if the request uses Signature Version 4, then following the redirect to the new hostname will result in a SignatureDoesNotMatch error because the hostname will be different. Back in the old days of Signature Version 2, the bucket name was included in the signature but the endpoint hostname itself was not, so the redirect to a different endpoint hostname would not invalidate the signature.

None of this would be a problem if boto were doing the right thing and using the correct regional endpoint to create the signed URL -- but for some reason, it uses the "global" (generic) endpoint -- which causes S3 to issue those redirects for the first few minutes of the bucket's lifetime, because DNS hasn't been updated, so the request misroutes to us-east-1 and gets redirected. That's why I suspected us-east-1 wouldn't exhibit the behavior.

This should be the default behavior, but it isn't; still, it seems like there should be a cleaner way to do this, automatically via configuration... and there may be... but I haven't found it in the documentation.

As a workaround, the client constructor accepts an endpoint_url argument, which seems to serve the purpose. As it turns out, s3.${region}.amazonaws.com is a valid endpoint for each S3 region, so they can be constructed from a region string.

s3_client = boto3.client('s3', region_name=region, endpoint_url=('https://s3.' + region + '.amazonaws.com'), config=...)

Long-time users of S3 may be suspicious of the claim that all regions support this, but it is accurate as of this writing. Originally, some regions formerly used a dash rather than a dot, e.g. s3-us-west-2.amazonaws.com and this is still valid in those older regions, but all regions now support the canonical form mentioned above.

user_16959060
Related