boto3: How to interract with DigitalOcean S3 Spaces when CDN is enabled

user_11049364 0
Viewed 1223

I'm working with DigitalOcean Spaces (S3 storage protocol) which has enabled CDN.

Any file on s3 can be accessed via direct URL in the given form:

https://my-bucket.fra1.digitaloceanspaces.com/<file_key>

If CDN is enabled, the file can be accessed via additional CDN URL:

https://my-bucket.fra1.cdn.digitaloceanspaces.com/<file_key>

where fra1 is a region_name.

When I'm using boto3 SDK for Python, the file URL is the following (generated by boto3):

https://fra1.digitaloceanspaces.com/my-bucket/<file_key>
# just note that bucket name is no more a domain part!

This format also works fine.

But, if CDN is enabled - file url causes an error:

EndpointConnectionError: Could not connect to the endpoint URL:  https://fra1.cdn.digitaloceanspaces.com/my-bucket/<file_key>

assuming the endpoint_url was changed from

default_endpoint=https://fra1.digitaloceanspaces.com

to

default_endpoint=https://fra1.cdn.digitaloceanspaces.com

How to connect to CDN with proper URL without getting an error? And why boto3 uses different URL format? Is any workaround can be applied in this case?

code:

s3_client = boto3.client('s3',
                            region_name=s3_configs['default_region'],
                            endpoint_url=s3_configs['default_endpoint'],
                            aws_access_key_id=s3_configs['bucket_access_key'],
                            aws_secret_access_key=s3_configs['bucket_secret_key'])

s3_client.download_file(bucket_name,key,local_filepath)

boto3 guide for DigitalOcean Spaces.

Here is what I've also tried but It didn't work:

UPDATE Based on @Amit Singh's answer:

As I mentioned before, I've already tried this trick with presigned URLs. I've got Urls like this

https://fra1.digitaloceanspaces.com/<my-bucket>/interiors/uploaded/images/07IRgHJ2PFhVqVrJDCIpzhghqe4TwK1cSSUXaC4T.jpeg?<presigned-url-params>

The bucket name appears after endpoint. I had to move It to domain-level manually:

https://<my-bucket>.fra1.cdn.digitaloceanspaces.com/interiors/uploaded/images/07IRgHJ2PFhVqVrJDCIpzhghqe4TwK1cSSUXaC4T.jpeg?<presigned-url-params>

With this URL I can now connect to Digital ocean, but another arror occures:

This XML file does not appear to have any style information associated with it. The document tree is shown below.
<Error>
<Code>SignatureDoesNotMatch</Code>
<RequestId>tx00000000000008dfdbc88-006005347c-604235a-fra1a</RequestId>
<HostId>604235a-fra1a-fra1</HostId>
</Error>

As a workaround I've tired to use signature s3v4:

    s3_client = boto3.client('s3',
                                 region_name=configs['default_region'],
                                 endpoint_url=configs['default_endpoint'],
                                 aws_access_key_id=configs['bucket_access_key'],
                                 aws_secret_access_key=configs['bucket_secret_key'],
                                 config= boto3.session.Config(signature_version='s3v4'))

but It still fails.

digital-ocean boto3 cdn amazon-s3 python
2 Answers

boto3 is a client library for Amazon S3 and not Digital Ocean Spaces. So, boto3 will not recognize the CDN URL fra1.cdn.digitaloceanspaces.com since it is provided by Digital Ocean and the URL with CDN is not one of the supported URI patterns. I don't fully understand how CDNs work internally, so my guess is there might be challenges with implementing this redirection to correct URL.

Now that that's clear, let's see how we can get a pre-signed CDN URL. Suppose, your CDN URL is https://fra1.cdn.digitaloceanspaces.com and your space name is my-space. We want to get a pre-signed URL for an object my-example-object stored in the space.

import os
import boto3
from botocore.client import Config

# Initialize the client
session = boto3.session.Session()
client = session.client('s3',
                        region_name='fra1',
                        endpoint_url='https://fra1.digitaloceanspaces.com', # Remove `.cdn` from the URL
                        aws_access_key_id=os.getenv('SPACES_KEY'),
                        aws_secret_access_key=os.getenv('SPACES_SECRET'),
                        config=Config(s3={'addressing_style': 'virtual'}))

# Get a presigned URL for object 
url = client.generate_presigned_url(ClientMethod='get_object',
                                    Params={'Bucket': 'my-space',
                                            'Key': 'my-example-object'},
                                    ExpiresIn=300)

print(url)

The pre-signed URL will look something like :

https://my-space.fra1.digitaloceanspaces.com/my-example-object?AWSAccessKeyId=EXAMPLE7UQOTHDTF3GK4&Content-Type=text&Expires=1580419378&Signature=YIXPlynk4BALXE6fH7vqbnwjSEw%3D

Add the cdn in between either manually or programmatically, in case you need to so that your final URL will become:

https://my-space.fra1.cdn.digitaloceanspaces.com/my-example-object?AWSAccessKeyId=EXAMPLE7UQOTHDTF3GK4&Content-Type=text&Expires=1580419378&Signature=YIXPlynk4BALXE6fH7vqbnwjSEw%3D

This is your CDN URL.

user_84916420

Based on @Amit Singh's answer, I've made an additional research of this issue.

Answers that helped me were found here and here.

To make boto3 presigned URLs work, I've made the following update to client and generate_presigned_url() params.

s3_client = boto3.client('s3',
                         region_name=configs['default_region'],
                         endpoint_url=configs['default_endpoint'],
                         aws_access_key_id=configs['bucket_access_key'],
                         aws_secret_access_key=configs['bucket_secret_key'],
                         config=boto3.session.Config(signature_version='s3v4', retries={
                             'max_attempts': 10,
                             'mode': 'standard'
                         },
                         s3={'addressing_style': "virtual"}, ))
...

response = s3_client.generate_presigned_url('get_object',
                                            Params={'Bucket': bucket_name,
                                                    'Key': object_name},
                                            ExpiresIn=3600,
                                            HttpMethod=None
                                            )

After that, .cdn domain part shoud be added after region name.

user_110493640
Related