Unknown spike in 'Approximate Age of Oldest Message Average' matrix AWS

user_1551285 0
Viewed 1112

I have received following alarm message daily at the same time from my Amazon SQS.

You are receiving this email because your Amazon CloudWatch Alarm "Old Messages in SQS" in the {my region} region has entered the ALARM state, because "Threshold Crossed: 1 out of the last 1 datapoints [183.0 (30/09/20 00:06:00)] was greater than or equal to the threshold (180.0) (minimum 1 datapoint for OK -> ALARM transition)." at "Wednesday 30 September, 2020 00:07:22 UTC".

Alarm Details:

  • Name: Old Messages in SQS
  • Description: Abc updates take too long. Check the processor and queue.
  • State Change: OK -> ALARM
  • Reason for State Change: Threshold Crossed: 1 out of the last 1 datapoints [183.0 (30/09/20 00:06:00)] was greater than or equal to the threshold (180.0) (minimum 1 datapoint for OK -> ALARM transition).
  • Timestamp: Wednesday 30 September, 2020 00:07:22 UTC

Threshold:

  • The alarm is in the ALARM state when the metric is GreaterThanOrEqualToThreshold 180.0 for 60 seconds.

Monitored Metric:

  • MetricNamespace: AWS/SQS
  • MetricName: ApproximateAgeOfOldestMessage
  • Period: 60 seconds
  • Statistic: Average
  • Unit: not specified

State Change Actions:

  • OK:
  • INSUFFICIENT_DATA:

So I checked in cloudwatch and see what is happening. So I identified that CPU utilization is going low at the same time of that instance which is used to process messages in SQS. So I decided that messages in SQS were increased due to server down.

SQS oldest message

CPU Utilization

But I can't identify why server is going down at the same time every day. I checked followings

  • EC2 snapshots - no automated schedules
  • RDS snapshots - no automated schedules at that time
  • Cron jobs in the server

Is there anyone who has this kind of experience will be highly appreciated to identify what the exact issue is.

amazon-cloudwatch amazon-sqs amazon-web-services amazon-ec2
1 Answers

This is super late to the party and I assume by now you either figured it out or you moved on. Sharing some thoughts for posterity.

The metric that went over threshold of 3 mins is age of the item on the SQS queue. The cause of the alert is not necessarily an instance processing messages going down but could be that the processing routine is blocked (waiting) on something external. This could be a network call that lasts long, or something like that.

During the waiting state CPU is not used that much so CPU utilization could be close to 0 if no other messages are being processed. Again, this is just a guess.

What I would do in this situation is:

  1. Add/check cloudwatch logs that are emitted within the processing routine and verify processing is not stuck.

  2. Check instance events / logs (by these I mean instance starting/stopping, maintenance, etc. to verify nothing external was affecting my processing instance)

  3. In a number of cases the answer could be an error on Amazon side so when stuck - check with Amazon support.

user_1771540
Related