Unable to validate the following destination configurations (S3 to SQS)

Question

I am trying to set up a workflow with serverless that creates a new S3 bucket, a new SQS queue and when an object is created in the S3 bucket, puts a messages on the queue and spins up a lambda once there are enough messages on the queue. I have the following in my resources block:

resources:
  Resources:
    AnalyticsQueue:
      Type: "AWS::SQS::Queue"
      Properties:
        QueueName: "my-queue"
    S3EventQueuePolicy:
      Type: AWS::SQS::QueuePolicy
      DependsOn: AnalyticsQueue
      Properties:
        PolicyDocument:
          Id: SQSPolicy
          Statement:
            - Effect: Allow
              Action: sqs:SendMessage:*
              Resource: !Ref AnalyticsQueue
        Queues:
          - !GetAtt AnalyticsQueue.Arn
    AnalyticsBucket:
      Type: AWS::S3::Bucket
      Properties:
        BucketName: "my-bucket"
        NotificationConfiguration:
          QueueConfigurations:
            - Event: s3:ObjectCreated:*
              Queue: !GetAtt AnalyticsQueue.Arn

When I try to deploy this I receive the following error:

An error occurred: AnalyticsBucket - Unable to validate the following destination configurations (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument; Request ID: E2A1F8BD6BEE6EF4;).

Some googling and I found that the issue is in the NotificationConfiguration block on the AnalyticsBucket. If I remove that whole sub-block, it deploys just fine but then obviously won't generate messages on the queue when objects get created.

Looking for a way to resolve this.

user_1748964 · Answer

A lot of AWS configuration allows you to connect services and they fail at runtime if they don't have permission, however S3 notification configuration does check some destinations for access.

In this case, you haven't allowed S3 to send messages to SQS.

It should be something like:

  PolicyDocument:
    Id: SQSPolicy
    Statement:
    - Sid: SQSEventPolicy
      Effect: Allow
      Principal: "*"
      Action: SQS:*
      Resource: "*"
      Condition:
        ArnLike:
          aws:SourceArn: arn:aws:s3:::*

user_15030095 · Answer

Adding this access policy worked for me:

{
  "Version": "2012-10-17",
  "Id": "example-ID",
  "Statement": [
    {
      "Sid": "example-statement-ID",
      "Effect": "Allow",
      "Principal": {
        "Service": "s3.amazonaws.com"
      },
      "Action": [
        "SQS:SendMessage"
      ],
      "Resource": "arn:aws:sqs:Region:account-id:queue-name",
      "Condition": {
        "ArnLike": {
          "aws:SourceArn": "arn:aws:s3:*:*:awsexamplebucket1"
        },
        "StringEquals": {
          "aws:SourceAccount": "bucket-owner-account-id"
        }
      }
    }
  ]
}

Source: https://docs.aws.amazon.com/AmazonS3/latest/userguide/grant-destinations-permissions-to-s3.html

user_19350961 · Answer

This tutorial helped me to solve the Unknown Error API response: https://www.youtube.com/watch?v=S7SFw8mMMTM

As a summary what needed to be done is to delete this part in the access policy:

"StringEquals": {
          "aws:SourceAccount": "bucket-owner-account-id"

so the access policy will look like this:

{
  "Version": "2012-10-17",
  "Id": "example-ID",
  "Statement": [
    {
      "Sid": "example-statement-ID",
      "Effect": "Allow",
      "Principal": {
        "Service": "s3.amazonaws.com"
      },
      "Action": [
        "SQS:SendMessage"
      ],
      "Resource": "arn:aws:sqs:Region:account-id:queue-name",
      "Condition": {
        "ArnLike": {
          "aws:SourceArn": "arn:aws:s3:*:*:awsexamplebucket1"
        }
      }
    }
  ]
}

Hope it helps

user_4526737 · Answer

I tried to create the event manually in the AWS console and got the same problem. It seems this problem occurs if we register an event to encrypted SQS. I tried to disable the SQS encryption to solve this problem. Maybe for encrypted SQS, further configuration is needed.

user_15613143 · Answer

This errror may be predominantly due to encryption enabled in the sqs queue. The solution is either disable encryption in sqs or else use an encryption key with proper permissions to key the encrypt/decrypt s3 notification.

user_7121954 · Answer

I was facing the same issue to give cross account access to publish S3 update message to my SQS. It can be referenced from https://docs.aws.amazon.com/AmazonS3/latest/userguide/grant-destinations-permissions-to-s3.html

I have a CDK package and the following worked for me And we can set up the policy.

    this..addToResourcePolicy(
        new iam.PolicyStatement({
            effect: Effect.ALLOW,
            principals: [new SecureServicePrincipal('s3.amazonaws.com')],
            actions: ['SQS:SendMessage'],
            resources: [this..queueArn],
            conditions: {
                ArnLike: {
                    'aws:SourceArn': ,
                },
                StringEquals: {
                    'aws:SourceAccount': ,
                }
            }
        })
    );

user_10399628 · Answer

The AWS:SourceAccount should be your Account ID (Top right of the page, clicking on your username)

"StringEquals": {
  "AWS:SourceAccount": "XXXXXXXXXXX"
}

See AWS Documentation

user_4905958 · Answer

While I would definitely check the policy on your SQS queue as the first thing, the second thing would be to open your browser's developer tools and look at the actual request that the console sends back.

In our case, the response identified a specific queue that no longer existed (even though the AWS console UI just showed the generic "some error happened" message.

Additionally, the queue causing the error was not the queue we were trying to target with our new rule - it turns out that all the event notifications on a bucket are managed as one package. So "adding" a new event notification is really updating the existing set of event notifications.

If your existing set of event notifications contains references to Queues or other targets that no longer exist, (in our case a QA environment that had been deleted long ago), the API request to update the the set of notifications fails, because it references queues that no longer exists. The solution is to first delete the event notification rules that reference non-existent targets, then you can add a new rule.

Unable to validate the following destination configurations (S3 to SQS)

8 Answers