Can I redirect alerts from a single DataDog monitor to different Slack channels?

user_16691082 0
Viewed 2852

So I've got a DataDog monitor and its query is roughly similar to:

trace-analytics("service:foo-service @http.status_code:403").rollup("count").last("10m") > 1000

and under "Notify your team" section of DD monitor I setup my team's channel on Slack and we receive quite a lot of alerts as a result.

That said, we'd be interested to redirect those errors to relevant teams (e.g., if the route /abc throws a 403 we'd like to post it to another #abc-team Slack channel, /cde to #cde-team etc). How can we do it? I was thinking I could write a Slack bot or something since I didn't find such an option on DataDog.

slack-api slack datadog
2 Answers

You would need some kinda of tag that is an exact string match on some part of the slack channel (so the tag would need to be route:abc not route:/abc)

then you would be able to write something like @slack-{{route.name}}-alerts. which automatically dereferences the tag and sends it out to the appropriate group.

monitor message description

https://www.datadoghq.com/blog/tagging-best-practices/#create-automatic-dynamic-alerts-for-your-team

If for some reason you can't get a tag that matches any part of your slack handle onto the data, you could make a giant if else conditional

{{#is_match "route.name" "/abc"}}
  @slack-abc
{{/is_match}}
{{#is_match "route.name" "/whoever"}}
  @slack-whoever
{{/is_match}}
{{#is_match "route.name" "/turtle"}}
  @slack-turtle
{{/is_match}}
{{^is_match "route.name" "/abc" "/whoever" "/turtle"}}
  please make sure that this endpoint is configured to forward to the proper team's channel inside this monitor
  @slack-backup-alert-channel
{{/is_match}}

https://docs.datadoghq.com/monitors/notifications/?tab=monitors#conditional-variables

the first option is much cleaner and scales way better obviously. but i've seen people use the second one.

user_26761080


I am not much aware about the DataDog functionality, but I can help you with Slack Bot.

The bot you can develop will have following flow:

  1. All the alerts will be sent to common channel #all-alerts
  2. The bot will also be invited to this channel
  3. Subscribe the bot to message event
  4. Process the received event(message), determine which team it belongs to.
  5. Post the processed message/alert to relevant team.
user_109943020
Related