CloudFormation add trigger for existing s3 bucket

user_12178717 0
Viewed 2124

My goal is to pack my lambda code which is invoked on each image upload to bucket, into CloudFormation template. I achieved so far to create new resources, and trigger from scratch, but I have existing bucket to which I need to add trigger and get errors in 2 cases:

  1. when I put creation of lambda and trigger configuration in one template and try to create stack as new resources - it says that bucket already exists
  2. when I move trigger to new file - and first create new resources like in 1., then I import existing resources into the stack - I get:

There was an error creating this change set

You have modified resources [ScaleImages, ScaleImagesRole] in your template that are not being imported. Update, create or delete operations cannot be executed during import operations.

My templates looks like:

  • lambda creation - new lambda and role - create stack with new resources
    {
      "AWSTemplateFormatVersion": "2010-09-09",
      "Resources": {
        "ScaleImages": {
          "Type": "AWS::Lambda::Function",
          "DeletionPolicy": "Retain",
          "Properties": {
            "FunctionName": "ScaleImages",
            "Handler": "index.handler",
            "Role": {
              "Fn::GetAtt": [
                "ScaleImagesRole",
                "Arn"
              ]
            },
            "Code": {
              "S3Bucket": "example-test",
              "S3Key": "example-resize.zip"
            },
            "Runtime": "nodejs12.x",
            "MemorySize": 1024,
            "Timeout": 300
          }
        },
        "ScaleImagesRole": {
          "Type": "AWS::IAM::Role",
          "DeletionPolicy": "Retain",
          "Properties": {
            "RoleName": "ScaleImagesRole",
            "AssumeRolePolicyDocument": {
              "Version": "2012-10-17",
              "Statement": [
                {
                  "Effect": "Allow",
                  "Principal": {
                    "Service": [
                      "lambda.amazonaws.com"
                    ]
                  },
                  "Action": [
                    "sts:AssumeRole"
                  ]
                }
              ]
            },
            "Path": "/",
            "Policies": [
              {
                "PolicyName": "AWSLambdaBasicExecutionRole",
                "PolicyDocument": {
                  "Version": "2012-10-17",
                  "Statement": [
                    {
                      "Effect": "Allow",
                      "Action": [
                        "logs:CreateLogGroup",
                        "logs:CreateLogStream",
                        "logs:PutLogEvents"
                      ],
                      "Resource": "*"
                    }
                  ]
                }
              },
              {
                "PolicyName": "AmazonS3FullAccess",
                "PolicyDocument": {
                  "Version": "2012-10-17",
                  "Statement": [
                    {
                      "Effect": "Allow",
                      "Action": "s3:*",
                      "Resource": [
                        "arn:aws:s3:::example-test",
                        "arn:aws:s3:::example-test/*",
                        "arn:aws:s3:::example-test-output",
                        "arn:aws:s3:::example-test-output/*"
                      ]
                    }
                  ]
                }
              }
            ]
          }
        }
      }
    }
  • adding trigger - bucket exists - Import resources
{
  "AWSTemplateFormatVersion": "2010-09-09",
  "Resources": {
    "PutOriginalImage": {
      "Type": "AWS::S3::Bucket",
      "DeletionPolicy": "Retain",
      "Properties": {
        "BucketName": "example-test",
        "NotificationConfiguration": {
          "LambdaConfigurations": [
            {
              "Event": "s3:ObjectCreated:Put",
              "Filter": {
                "S3Key": {
                  "Rules": [
                    {
                      "Name": "prefix",
                      "Value": "original2/"
                    }
                  ]
                }
              },
              "Function": {
                "Fn::GetAtt": [
                  "ScaleImages",
                  "Arn"
                ]
              }
            }
          ]
        }
      }
    }
  }
}

In last one I tried also "Function": "ScaleImages", but in both cases I had same error about:

modified resources [ScaleImages, ScaleImagesRole] in your template

Can somebody shed some light on what I'm doing wrong?

amazon-cloudformation aws-lambda amazon-web-services amazon-s3
2 Answers

You have to do it in stages:

1. Create new stack

No bucket yet, just stack with your function and lambda permissions which you are missing.

    {
      "AWSTemplateFormatVersion": "2010-09-09",
      "Resources": {
        "ScaleImages": {
          "Type": "AWS::Lambda::Function",
          "Properties": {
            "FunctionName": "ScaleImages",
            "Handler": "index.handler",
            "Role": {
              "Fn::GetAtt": [
                "ScaleImagesRole",
                "Arn"
              ]
            },
            "Code": {
              "S3Bucket": "example-test",
              "S3Key": "example-resize.zip"
            },
            "Runtime": "nodejs12.x",
            "MemorySize": 1024,
            "Timeout": 300
          }
        },
        "ScaleImagesRole": {
          "Type": "AWS::IAM::Role",
          "Properties": {
            "RoleName": "ScaleImagesRole",
            "AssumeRolePolicyDocument": {
              "Version": "2012-10-17",
              "Statement": [
                {
                  "Effect": "Allow",
                  "Principal": {
                    "Service": [
                      "lambda.amazonaws.com"
                    ]
                  },
                  "Action": [
                    "sts:AssumeRole"
                  ]
                }
              ]
            },
            "Path": "/",
            "Policies": [
              {
                "PolicyName": "AWSLambdaBasicExecutionRole",
                "PolicyDocument": {
                  "Version": "2012-10-17",
                  "Statement": [
                    {
                      "Effect": "Allow",
                      "Action": [
                        "logs:CreateLogGroup",
                        "logs:CreateLogStream",
                        "logs:PutLogEvents"
                      ],
                      "Resource": [
                        "arn:aws:s3:::example-test",
                        "arn:aws:s3:::example-test/*",
                        "arn:aws:s3:::example-test-output",
                        "arn:aws:s3:::example-test-output/*"
                      ]
                    }
                  ]
                }
              },
              {
                "PolicyName": "AmazonS3FullAccess",
                "PolicyDocument": {
                  "Version": "2012-10-17",
                  "Statement": [
                    {
                      "Effect": "Allow",
                      "Action": "s3:*",
                      "Resource": "*"
                    }
                  ]
                }
              }
            ]
          }
        },


"s3Permission": {
    "Type": "AWS::Lambda::Permission",
    "Properties": {
        "FunctionName": {
            "Fn::GetAtt": [
                "ScaleImages",
                "Arn"
            ]
        },
        "Action": "lambda:InvokeFunction",
        "Principal": "s3.amazonaws.com",
        "SourceAccount": {
            "Ref": "AWS::AccountId"
        }
    }
}



      }
    }

2. Import bucket to existing stack

Use Import resources into stack option and upload stack using this template. It adds bucket, but no notifications yet

    {
      "AWSTemplateFormatVersion": "2010-09-09",
      "Resources": {
        "ScaleImages": {
          "Type": "AWS::Lambda::Function",
          "Properties": {
            "FunctionName": "ScaleImages",
            "Handler": "index.handler",
            "Role": {
              "Fn::GetAtt": [
                "ScaleImagesRole",
                "Arn"
              ]
            },
            "Code": {
              "S3Bucket": "example-test",
              "S3Key": "example-resize.zip"
            },
            "Runtime": "nodejs12.x",
            "MemorySize": 1024,
            "Timeout": 300
          }
        },
        "ScaleImagesRole": {
          "Type": "AWS::IAM::Role",
          "Properties": {
            "RoleName": "ScaleImagesRole",
            "AssumeRolePolicyDocument": {
              "Version": "2012-10-17",
              "Statement": [
                {
                  "Effect": "Allow",
                  "Principal": {
                    "Service": [
                      "lambda.amazonaws.com"
                    ]
                  },
                  "Action": [
                    "sts:AssumeRole"
                  ]
                }
              ]
            },
            "Path": "/",
            "Policies": [
              {
                "PolicyName": "AWSLambdaBasicExecutionRole",
                "PolicyDocument": {
                  "Version": "2012-10-17",
                  "Statement": [
                    {
                      "Effect": "Allow",
                      "Action": [
                        "logs:CreateLogGroup",
                        "logs:CreateLogStream",
                        "logs:PutLogEvents"
                      ],
                      "Resource": [
                        "arn:aws:s3:::example-test",
                        "arn:aws:s3:::example-test/*",
                        "arn:aws:s3:::example-test-output",
                        "arn:aws:s3:::example-test-output/*"
                      ]
                    }
                  ]
                }
              },
              {
                "PolicyName": "AmazonS3FullAccess",
                "PolicyDocument": {
                  "Version": "2012-10-17",
                  "Statement": [
                    {
                      "Effect": "Allow",
                      "Action": "s3:*",
                      "Resource": "*"
                    }
                  ]
                }
              }
            ]
          }
        },


"s3Permission": {
    "Type": "AWS::Lambda::Permission",
    "Properties": {
        "FunctionName": {
            "Fn::GetAtt": [
                "ScaleImages",
                "Arn"
            ]
        },
        "Action": "lambda:InvokeFunction",
        "Principal": "s3.amazonaws.com",
        "SourceAccount": {
            "Ref": "AWS::AccountId"
        }
    }
}
,
    "PutOriginalImage": {
      "Type": "AWS::S3::Bucket",
      "DeletionPolicy": "Retain",
      "Properties": {
        "BucketName": "example-test"
      }
    }

      }
    }

3. Update stack

Update stack by adding notifications to the bucket. Use the following template:

    {
      "AWSTemplateFormatVersion": "2010-09-09",
      "Resources": {
        "ScaleImages": {
          "Type": "AWS::Lambda::Function",
          "Properties": {
            "FunctionName": "ScaleImages",
            "Handler": "index.handler",
            "Role": {
              "Fn::GetAtt": [
                "ScaleImagesRole",
                "Arn"
              ]
            },
            "Code": {
              "S3Bucket": "example-test",
              "S3Key": "example-resize.zip"
            },
            "Runtime": "nodejs12.x",
            "MemorySize": 1024,
            "Timeout": 300
          }
        },
        "ScaleImagesRole": {
          "Type": "AWS::IAM::Role",
          "Properties": {
            "RoleName": "ScaleImagesRole",
            "AssumeRolePolicyDocument": {
              "Version": "2012-10-17",
              "Statement": [
                {
                  "Effect": "Allow",
                  "Principal": {
                    "Service": [
                      "lambda.amazonaws.com"
                    ]
                  },
                  "Action": [
                    "sts:AssumeRole"
                  ]
                }
              ]
            },
            "Path": "/",
            "Policies": [
              {
                "PolicyName": "AWSLambdaBasicExecutionRole",
                "PolicyDocument": {
                  "Version": "2012-10-17",
                  "Statement": [
                    {
                      "Effect": "Allow",
                      "Action": [
                        "logs:CreateLogGroup",
                        "logs:CreateLogStream",
                        "logs:PutLogEvents"
                      ],
                      "Resource": [
                        "arn:aws:s3:::example-test",
                        "arn:aws:s3:::example-test/*",
                        "arn:aws:s3:::example-test-output",
                        "arn:aws:s3:::example-test-output/*"
                      ]
                    }
                  ]
                }
              },
              {
                "PolicyName": "AmazonS3FullAccess",
                "PolicyDocument": {
                  "Version": "2012-10-17",
                  "Statement": [
                    {
                      "Effect": "Allow",
                      "Action": "s3:*",
                      "Resource": "*"
                    }
                  ]
                }
              }
            ]
          }
        }

,

"s3Permission": {
    "Type": "AWS::Lambda::Permission",
    "Properties": {
        "FunctionName": {
            "Fn::GetAtt": [
                "ScaleImages",
                "Arn"
            ]
        },
        "Action": "lambda:InvokeFunction",
        "Principal": "s3.amazonaws.com",
        "SourceAccount": {
            "Ref": "AWS::AccountId"
        }
    }
},



            "PutOriginalImage": {
      "Type": "AWS::S3::Bucket",
      "DeletionPolicy": "Retain",
      "Properties": {
        "BucketName": "example-test",

        "NotificationConfiguration": {
          "LambdaConfigurations": [
            {
              "Event": "s3:ObjectCreated:Put",
              "Filter": {
                "S3Key": {
                  "Rules": [
                    {
                      "Name": "prefix",
                      "Value": "original2/"
                    }
                  ]
                }
              },
              "Function": {
                "Fn::GetAtt": [
                  "ScaleImages",
                  "Arn"
                ]
              }
            }
          ]
        }


      }
    }



      }
    }
user_2488230

A slightly different approach that gets you going in one shot without following 3 steps. I have been through some tough times while importing existing resources in Cloudformation, I would handle the complexity in the lambda via a custom resource

                s3 = boto3.resource('s3')
                
                def lambda_handler(event, context):
                    print("Received event: " + json.dumps(event, indent=2))
                    responseData={}
                    try:
                        if event['RequestType'] == 'Delete':
                            print("Request Type:",event['RequestType'])
                            Bucket=event['ResourceProperties']['Bucket']
                            delete_notification(Bucket)
                            print("Sending response to custom resource after Delete")
                        elif event['RequestType'] == 'Create' or event['RequestType'] == 'Update':
                            print("Request Type:",event['RequestType'])
                            LambdaArn=event['ResourceProperties']['LambdaArn']
                            Bucket=event['ResourceProperties']['Bucket']
                            add_notification(LambdaArn, Bucket)
                            responseData={'Bucket':Bucket}
                            print("Sending response to custom resource")
                        responseStatus = 'SUCCESS'
                    except Exception as e:
                        print('Failed to process:', e)
                        responseStatus = 'FAILURE'
                        responseData = {'Failure': 'Something bad happened.'}
                    cfnresponse.send(event, context, responseStatus, responseData)
    
                def add_notification(LambdaArn, Bucket):
                    bucket_notification = s3.BucketNotification(Bucket)
                    response = bucket_notification.put(
                      NotificationConfiguration={
                        'LambdaFunctionConfigurations': [
                          {
                              'LambdaFunctionArn': LambdaArn,
                              'Events': [
                                  's3:ObjectCreated:*'
                              ]
                          }
                        ]
                      }
                    )
                    print("Put request completed....")
                  
                def delete_notification(Bucket):
                    bucket_notification = s3.BucketNotification(Bucket)
                    response = bucket_notification.put(
                        NotificationConfiguration={}
                    )
                    print("Delete request completed....")

Full template and solution can be found here

Note : There is alaready an open issue on AWS CloudFormation Repo on github . Orignally came from the serverless folks

user_22463450
Related