What is the best possible way to send alerts to slack on lambda function failures?

user_10689474 0
Viewed 769

As far I know there have been multiple approaches to do this. Some approaches that I know are

  1. Monitoring cloudwatch events and trigger a cloud watch alarm if lambda fails. Using cloud watch alarm we can trigger sns/slack.

Using the first approach, I can't use a single alarm to monitor my lambdas. Because If I have 100s of lambdas it is difficult to identify which lambda is actually failed.

  1. Using Destination. we can trigger sns or another lambda function asynchronously on failure.

Are there any other ways to get notified on lambda failures? If so, which is the best approach?

amazon-cloudwatch amazon-sns aws-lambda amazon-web-services
2 Answers

Not sure what you mean by "I can't use a single alarm to monitor my lambdas", generally you have one single alarm per lambda. This is a good practice since every lambda could have a different set of characteristics so your monitoring metrics can vary from one to the other. This also helps you since every alarm can have a different name like service-stage-region-alarm-myFunction so it is easily identified. Generate one alarm per lambda can easily be done if you are using frameworks like serverless or SAM and is a reasonable task with tools like terraform or cloudformation.

Second approach could be a valid one if you want to fire one notification per error. This probably is not what you want since under heavy traffic you don't want to receive hundreds of messages with the same error. Also it does not work for synchronous lambdas (like the ones attached to APIGateway).

user_24514250

@RoberMP listed very good points. To extend him;

After creating individual alarms with different configurations, thresholds and limits you may not want to trigger SNS/Slack after each failure.

If you don't want to create too much noise with those alarms, you may combine your first approach with composite alarms.

Using composite alarms can help you reduce alarm noise. If you set up a composite alarm to notify you of state changes, but set up the underlying metric alarms to not send notifications themselves, you will be notified only when the alarm state of the composite alarm changes.

user_21889220
Related