Does AWS CPP S3 SDK support "Transfer acceleration"

user_926264 0
Viewed 445

I enabled "Transfer acceleration" on my bucket. But I dont see any improvement in speed of Upload in my C++ application. I have waited for more than 20 minutes that is mentioned in AWS Documentation.

Does the SDK support "Transfer acceleration" by default or is there a run time flag or compiler flag? I did not spot anything in the SDK code.

thanks

aws-sdk-cpp amazon-web-services amazon-s3 c++
3 Answers

What I did to enable a (working) transfer acceleration:

  • set in the bucket configuration on the AWS panel "Transfer Acceleration" to enabled.

  • add to the IAM user that I use inside my C++ application the permission s3::PutAccelerateConfiguration

  • Add the following code to the s3 transfer configuration (bucket_ is your bucket name, the final URL must match the one shown in the AWS panel "Transfer Acceleration"):

    Aws::Client::ClientConfiguration config;
    /* other configuration options */
    config.endpointOverride = bucket_ + ".s3-accelerate.amazonaws.com";
  • Ask for acceleration to the bucket before transfer... (docs in here )
    auto s3Client = Aws::MakeShared<Aws::S3::S3Client>("Uploader",
            Aws::Auth::AWSCredentials(id_, key_), config);
    Aws::S3::Model::PutBucketAccelerateConfigurationRequest bucket_accel;
    bucket_accel.SetAccelerateConfiguration( 
        Aws::S3::Model::AccelerateConfiguration().WithStatus(
        Aws::S3::Model::BucketAccelerateStatus::Enabled));
    bucket_accel.SetBucket(bucket_);
    s3Client->PutBucketAccelerateConfiguration(bucket_accel);

You can check in the detailed logs of the AWS sdk that your code is using the accelerated entrypoint and you can also check that before the transfer start there is a call to /?accelerate (info)

user_15630370

What worked for me:

  1. Enabling S3 Transfer Acceleration within AWS console

  2. When configuring the client, only utilize the accelerated endpoint service:

    clientConfig->endpointOverride = "s3-accelerate.amazonaws.com";

@gabry - your solution was extremely close, I think the reason it wasn't working for me was perhaps due to SDK changes since originally posted as the change is relatively small. Or maybe because I am constructing put object templates for requests used with the transfer manager.

Looking through the logs (Debug level) the SDK automatically concatenates the bucket used in transferManager::UploadFile() with the overridden endpoint. I was getting unresolved host errors as the requested host looked like:

[DEBUG] host: myBucket.myBucket.s3-accelerate.amazonaws.com

This way I could still keep the same S3_BUCKET macro name while only selectively calling this when instantiating a new configuration for upload.

e.g.

<<
...
  auto putTemplate = new Aws::S3::Model::PutObjectRequest();
  putTemplate->SetStorageClass(STORAGE_CLASS);
  transferConfig->putObjectTemplate = *putTemplate;

  auto multiTemplate = new Aws::S3::Model::CreateMultipartUploadRequest();
  multiTemplate->SetStorageClass(STORAGE_CLASS);
  transferConfig->createMultipartUploadTemplate = *multiTemplate;

  transferMgr = Aws::Transfer::TransferManager::Create(*transferConfig);

  auto transferHandle = transferMgr->UploadFile(localFile, S3_BUCKET, s3File);
  transferMgr = Aws::Transfer::TransferManager::Create(*transferConfig);


...

 >>
user_175352920
Related