Is there a way to list more than 1000 buckets using aws sdk C++?

user_1601656 0
Viewed 123

The AWS SDK page shows this example:

Aws::S3::S3Client s3_client;
Aws::S3::Model::ListBucketsOutcome outcome = s3_client.ListBuckets();

This, however, allows returning up to 1000 buckets ONLY!

In our organization we have more than 1k buckets. boto3 or java interface using ECS allows me to do pagination. I can find NOTHING, however, for C++ and I was already digging in the dark corners of the Internet.

Anyone has any idea how to do that pagination in C++ since ListBuckets() does not get any request as the argument?

NOTE: I am not looking for workarounds like executing a boto script or jni within my C++ to solve that list buckets issue. I am interested to find a proper way to use SDK, which I personally, for unknown reason to me does not exist

aws-sdk c++
1 Answers

I was having the same problem and I came across a solution for java here (https://stackoverflow.com/a/15352712/1856251) There are a number of solution there but the one in the link showed me the direction I needed.

It seems that for listing objects there is the concept of a Marker. This marker refers to the key to start with when listing. When it is empty the first key in the bucket. If you set this to the last key returned from the previous call to ListObjects then the next call will start from there. To do that you can call the member function.

void Aws::S3::Model::ListObjectsRequest::SetMarker( Aws::String &&  value)

More info can be found:
https://sdk.amazonaws.com/cpp/api/0.14.3/class_aws_1_1_s3_1_1_model_1_1_list_objects_request.html#a72bef4f7da7f91661a7642da7dc3aa36

Here is the code.


void myListObjects(const Aws::String& bucketName,
    const Aws::String& region)
{
    Aws::Client::ClientConfiguration config;
    if (!region.empty())
    {
        config.region = region;
    }

    Aws::S3::S3Client s3_client(config);
    Aws::S3::Model::ListObjectsRequest request;
    request.WithBucket(bucketName);
    std::string  prelastMarker, lastMarker;

    std::cout << "Objects in bucket '" << bucketName << "': "
        << std::endl << std::endl;
    do {
        auto outcome = s3_client.ListObjects(request);
        prelastMarker = lastMarker;
        if (outcome.IsSuccess())
        {
            Aws::Vector<Aws::S3::Model::Object> keyList =
                outcome.GetResult().GetContents();
            for (Aws::S3::Model::Object& object : keyList)
            {
                std::cout << object.GetKey() << std::endl;
            }
            lastMarker = keyList[keyList.size() - 1].GetKey();//outcome.GetResult().GetNextMarker();
            request.SetMarker(lastMarker.c_str());
            outcome = s3_client.ListObjects(request);
        }
    } while (prelastMarker != lastMarker);
}
user_18562510
Related