AWS: Permissions for exporting logs from Cloudwatch to Amazon S3

user_16805666 0
Viewed 6791

I am trying to export logs from one of my CloudWatch log groups into Amazon S3, using AWS console.

I followed the guide from AWS documentation but with little success. My organization does not allow me to manage IAM roles/policies, however I was able to find out that my role is allowed all log-related operations (logs:* on all resources within the account).

Currently, I am stuck on the following error message:

Could not create export task. PutObject call on the given bucket failed. Please check if CloudWatch Logs has been granted permission to perform this operation.

My bucket policy is set in the following way:

{
    [
        ...
        {
            "Sid": "Cloudwatch Log Export 1",
            "Effect": "Allow",
            "Principal": {
                "Service": "logs.eu-central-1.amazonaws.com"
            },
            "Action": "s3:GetBucketAcl",
            "Resource": "arn:aws:s3:::my-bucket"
        },
        {
            "Sid": "Cloudwatch Log Export 2",
            "Effect": "Allow",
            "Principal": {
                "Service": "logs.eu-central-1.amazonaws.com"
            },
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::my-bucket/*"
        }
    ]
}

Prior to editing bucket policy, my error message had been

Could not create export task. GetBucketAcl call on the given bucket failed. Please check if CloudWatch Logs has been granted permission to perform this operation.

but editing the bucket policy fixed that. I would expect allowing PutObject to do the same, but this has not been the case.

Thank you for help.

amazon-cloudwatch amazon-iam amazon-web-services amazon-s3
4 Answers

Ensure when exporting the data you configure the following aptly

S3 bucket prefix - optional This would be the object name you want to use to store the logs.

While creating the policy for PutBucket, you must ensure the object/prefix is captured adequately. See the diff for the PutBucket statement Resource:


 {
   "Version": "2012-10-17",
   "Statement": [
     {
         "Action": "s3:GetBucketAcl",
         "Effect": "Allow",
         "Resource": "arn:aws:s3:::my-exported-logs",
         "Principal": { "Service": "logs.us-east-2.amazonaws.com" }
     },
     {
         "Action": "s3:PutObject" ,
         "Effect": "Allow",
-        "Resource": "arn:aws:s3:::my-exported-logs/*",
+        "Resource": "arn:aws:s3:::my-exported-logs/**_where_i_want_to_store_my_logs_***",
         "Condition": { "StringEquals": { "s3:x-amz-acl": "bucket-owner-full-control" } },
         "Principal": { "Service": "logs.us-east-2.amazonaws.com" }
     }
   ]
 }
user_43673800

Please check this guide Export log data to Amazon S3 using the AWS CLI Policy's looks like the document that you share but slight different. Assuming that you are doing this in same account and same region, please check that you are placing the right region ( in this example is us-east-2)

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": "s3:GetBucketAcl",
            "Effect": "Allow",
            "Resource": "arn:aws:s3:::my-exported-logs",
            "Principal": { "Service": "logs.us-east-2.amazonaws.com" }
        },
        {
            "Action": "s3:PutObject" ,
            "Effect": "Allow",
            "Resource": "arn:aws:s3:::my-exported-logs/*",
            "Condition": { "StringEquals": { "s3:x-amz-acl": "bucket-owner-full-control" } },
            "Principal": { "Service": "logs.us-east-2.amazonaws.com" }
        }
    ]
}

I think that bucket owner full control is not the problem here, the only chance is the region. Anyway, take a look to the other two examples in case that you were in different accounts/ using role instead user.

This solved my issue, that was the same that you mention.

user_75264500

One thing to check is your encryption settings. According to https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/S3ExportTasksConsole.html

Exporting log data to Amazon S3 buckets that are encrypted by AWS KMS is not supported.

Amazon S3-managed keys (SSE-S3) bucket encryption might solve your problem. If you use SSE-KMS, Cloudwatch can't access your encryption key in order to properly encrypt the objects as they are put into the bucket.

user_26863460

I had the same situation and what worked for me is to add the bucket name itself as a resource in the Allow PutObject Sid, like:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowLogsExportGetBucketAcl",
            "Effect": "Allow",
            "Principal": {
                "Service": "logs.eu-west-1.amazonaws.com"
            },
            "Action": "s3:GetBucketAcl",
            "Resource": "arn:aws:s3:::my-bucket"
        },
        {
            "Sid": "AllowLogsExportPutObject",
            "Effect": "Allow",
            "Principal": {
                "Service": "logs.eu-west-1.amazonaws.com"
            },
            "Action": "s3:PutObject",
            "Resource": [
                "my-bucket",
                "my-bucket/*"
            ]
        }
    ]
}

I also believe that all the other answers are relevant, especially using the time in milliseconds.

user_3481620
Related