The AWS CloudTrail logs are being stored into an S3 bucket in the Logs Account. The trail that generates these logs is in the Management Account. I would like the CloudTrail logs to be visible in CloudWatch in my Security Account.
I am (mostly) using the console, and I have set up various EventBridge permissions and Roles and so forth, detailed below. However, whenever I try to enable CloudWatch logs in the trail I get the error An internal error occurred. Refresh the page, and retry.
How do I make the CloudTrail logs visible in a log group in CloudWatch in my Security account?
Account and organisation structure is relatively simple - four accounts, with no SCP applied:
- Management Account
- Infrastructure Account
- Security Account
- Logs Account
The CloudTrail trail in the management account is storing the logs in the Logs Account bucket, this is enabled via a bucket policy (this is the template, placeholder values replaced by correct bucket name, management account id, organisation id, trail name, region):
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AWSCloudTrailAclCheck20150319",
"Effect": "Allow",
"Principal": {
"Service": [
"cloudtrail.amazonaws.com"
]
},
"Action": "s3:GetBucketAcl",
"Resource": "arn:aws:s3:::BUCKET"
},
{
"Sid": "AWSCloudTrailWrite20150319",
"Effect": "Allow",
"Principal": {
"Service": [
"cloudtrail.amazonaws.com"
]
},
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::BUCKET/AWSLogs/MANAGEMENT_ACCOUNT_ID/*/*",
"Condition": {
"StringEquals": {
"s3:x-amz-acl": "bucket-owner-full-control",
"aws:SourceArn": "arn:aws:cloudtrail:REGION:MANAGEMENT_ACCOUNT_ID:trail/TRAIL_NAME"
}
}
},
{
"Sid": "AWSCloudTrailWrite20150319",
"Effect": "Allow",
"Principal": {
"Service": [
"cloudtrail.amazonaws.com"
]
},
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::BUCKET/AWSLogs/ORGANISATION_ID/*",
"Condition": {
"StringEquals": {
"s3:x-amz-acl": "bucket-owner-full-control",
"aws:SourceArn": "arn:aws:cloudtrail:REGION:MANAGEMENT_ACCOUNT_ID:trail/TRAIL_NAME"
}
}
}
]
}
The Security Account EventBridge default bus has the following resource permissions policy on it (this is the template, placeholder values replaced by security account id, organisation id, region):
{
"Version": "2012-10-17",
"Statement": [{
"Sid": "allow_all_accounts_from_organization_to_put_events",
"Effect": "Allow",
"Principal": "*",
"Action": "events:PutEvents",
"Resource": "arn:aws:events:REGION:SECURITY_ACCOUNT_ID:event-bus/default",
"Condition": {
"StringEquals": {
"aws:PrincipalOrgID": "ORGANISATION_ID"
}
}
}]
}
The security account also has a CloudWatch log group with name "cloudtrail-logs".
In the Management Account, there is the TRAIL_NAME CloudTrail trail and I have created the CloudTrailForCloudWatchLogs-TRAIL_NAME role with the following permissions policy attached:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AWSCloudTrailCreateLogStream1",
"Effect": "Allow",
"Action": [
"logs:CreateLogStream"
],
"Resource": [
"arn:aws:logs:REGION:SECURITY_ACCOUNT_ID:log-group:LOG_GROUP:log-stream:MANAGEMENT_ACCOUNT_ID_CloudTrail_REGION*",
"arn:aws:logs:REGION:SECURITY_ACCOUNT_ID:log-group:LOG_GROUP:log-stream:ORGANISATION_ID*"
]
},
{
"Sid": "AWSCloudTrailPutLogEvents2",
"Effect": "Allow",
"Action": [
"logs:PutLogEvents"
],
"Resource": [
"arn:aws:logs:REGION:SECURITY_ACCOUNT_ID:log-group:LOG_GROUP:log-stream:MANAGEMENT_ACCOUNT_ID_CloudTrail_REGION*",
"arn:aws:logs:REGION:SECURITY_ACCOUNT_ID:log-group:LOG_GROUP:log-stream:ORGANISATION_ID*"
]
}
]
}
In order to try and enable CloudTrail into CloudWatch I do the following:
In the management account, go to CloudTrail, click on the TRAIL_NAME trail and then click on "Edit" by the CloudWatch Logs (this currently says "CloudWatch logs is not configured for this trail").
In the resulting edit screen:
- Check "CloudWatch logs enabled"
- Specify an existing log group: cloudtrail-logs
- Specify an existing IAM role: CloudTrailForCloudWatchLogs-TRAIL_NAME
- At the bottom of this edit form there is a "Policy document" dropdown, expanding that shows the following policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AWSCloudTrailCreateLogStream2014110",
"Effect": "Allow",
"Action": [
"logs:CreateLogStream"
],
"Resource": [
"arn:aws:logs:REGION:MANAGEMENT_ACCOUNT_ID:log-group:LOG_GROUP:log-stream:MANAGEMENT_ACCOUNT_ID_CloudTrail_REGION*"
]
},
{
"Sid": "AWSCloudTrailPutLogEvents20141101",
"Effect": "Allow",
"Action": [
"logs:PutLogEvents"
],
"Resource": [
"arn:aws:logs:REGION:MANAGEMENT_ACCOUNT_ID:log-group:LOG_GROUP:log-stream:MANAGEMENT_ACCOUNT_ID_CloudTrail_REGION*"
]
}
]
}
- Clicking the "Save Changes" button at the bottom of the Edit form generates the error message
An internal error occurred. Refresh the page, and retry.. If I go to IAM I will also find a new policy matching exactly the one in step (3) just above has been created and attached to the CloudTrailForCloudWatchLogs-TRAIL_NAME role.
Specific resources that have helped me this far (these links will probably eventually go stale so I've provided detail above):
- Creating the trail and getting that working:
- Setting up the EventBridge, Management Account role:
- https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-required-policy-for-cloudwatch-logs.html
- https://docs.aws.amazon.com/awscloudtrail/latest/userguide/monitor-cloudtrail-log-files-with-cloudwatch-logs.html
- https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/CloudWatchEvents-CrossAccountEventDelivery.html#ReceivingEventsFromAnotherAccount (note that these instructions are slightly out of date because they refer to CloudWatch event bus and this is now handled by EventBridge)
- And this StackOverflow question is close to this one, but not quite: it is about collecting CloudWatch logs from multiple accounts into one account: Cloudtrail to Cloudwatch to other account
What about doing it via the CLI / update 2021-12-21
Having already configured the EventBus in the SecurityAccount to accept events as described above, I then used the instructions here. This also doesn't work, but fails with a more informative error message: An error occurred (InvalidCloudWatchLogsLogGroupArnException) when calling the UpdateTrail operation: You must specify a log group that is owned by this account..
This suggests that it isn't possible for CloudWatch in Account B to monitor the logs written by CloudTrail in account A, at least not directly.
I am now investigating whether or not I can share the CloudTrail logs from the Management Account into the Security Account and so create the CloudWatch monitoring that way (starting here).
In more detail, what I have done:
aws iam create-role --role-name ROLE_NAME --assume-role-policy-document file://path/to/cloudtrail_assume_role.json
aws iam put-role-policy --role-name ROLE_NAME --policy-name cloudtrail-for-cloudwatch-policy --policy-document file://path/to/role-policy-document.json
aws cloudtrail update-trail --name TRAIL_NAME --cloud-watch-logs-log-group-arn LOG_GROUP_ARN --cloud-watch-logs-role-arn ROLE_ARN
Where cloudtrail_assume_role.json looks like this:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "cloudtrail.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
And role-policy-document.json looks like this:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AWSCloudTrailCreateLogStream1",
"Effect": "Allow",
"Action": [
"logs:CreateLogStream"
],
"Resource": [
"arn:aws:logs:REGION:SECURITY_ACCOUNT_ID:log-group:LOG_GROUP:log-stream:MANAGEMENT_ACCOUNT_ID_CloudTrail_REGION*",
"arn:aws:logs:REGION:SECURITY_ACCOUNT_ID:log-group:LOG_GROUP:log-stream:ORGANISATION_ID_*"
]
},
{
"Sid": "AWSCloudTrailPutLogEvents2",
"Effect": "Allow",
"Action": [
"logs:PutLogEvents"
],
"Resource": [
"arn:aws:logs:REGION:SECURITY_ACCOUNT_ID:log-group:LOG_GROUP:log-stream:MANAGEMENT_ACCOUNT_ID_CloudTrail_REGION*",
"arn:aws:logs:REGION:SECURITY_ACCOUNT_ID:log-group:LOG_GROUP:log-stream:ORGANISATION_ID_*"
]
}
]
}