Can I trigger lambda only when message count reaches target value in SQS?

user_6782365 0
Viewed 936

I'm writing AWS Lambda to read message from SQS and then save some messages to S3 after filtering. Since 80% messages will be filtered out, I decided write S3 once for 100000 SQS messages.

Is it possible to trigger the Lambda only when the messages in SQS reach 10000?

amazon-sqs aws-lambda amazon-web-services amazon-s3
1 Answers

It's possible with help of AWS CloudWatch.

You could configure an AWS CloudWatch Alarm which triggers an "AlarmAction" as soon as your SQS queue got 100000 visible messages. In case of an "Alarm" you are notifying a SNS Topic which then triggers your AWS Lambda.

If you are using AWS CloudFormation it might look as the following:

AWSTemplateFormatVersion: 2010-09-09
Resources:
  Queue:
    Type: AWS::SQS::Queue

  QueueVisibleMessagesTopic:
    Type: AWS::SNS::Topic

  LambdaExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole

  Lambda:
    Type: AWS::Lambda::Function
    Properties:
      Handler: handler.index
      Role: !GetAtt LambdaExecutionRole.Arn
      Runtime: nodejs8.10
      MemorySize: 128
      Timeout: 10

  LambdaSubscription:
    Type: AWS::SNS::Subscription
    Properties:
      Endpoint: !GetAtt Lambda.Arn
      Protocol: lambda
      TopicArn: !Ref QueueVisibleMessagesTopic

  LambdaSubscriptionPermissions:
    Type: AWS::Lambda::Permission
    Properties:
      FunctionName: !GetAtt Lambda.Arn
      Action: lambda:InvokeFunction
      Principal: sns.amazonaws.com
      SourceArn: !Ref QueueVisibleMessagesTopic

  QueueVisibleMessagesAlarm:
    Type: AWS::CloudWatch::Alarm
    Properties:
      Namespace: AWS/SQS
      MetricName: ApproximateNumberOfMessagesVisible
      Dimensions:
        - Name: QueueName
          Value: !GetAtt Queue.QueueName
      Statistic: Sum
      Period: 300
      EvaluationPeriods: 1
      ComparisonOperator: GreaterThanOrEqualToThreshold
      Threshold: 100000
      AlarmActions:
        - !Ref QueueVisibleMessagesTopic
user_31156070
Related