I am using fluentd-cloudwatch to forward logs from Kubernetes to cloudwatch. All logs are forwarded to the same log group, but each container has its own log stream. I have some filter metrics and alarms based these metrics. For example, one filter metric could be "xxx connection error". Every time the error appears in the log, I will receive an notification. That's all working. But the problem is that I couldn't tell which log stream the error is from as I wanted to know which container generated that error.
1 Answers
If you have an active alarm, you can see which logstream triggered it by navigating to the Cloudwatch dashboard, select the specific Alarm. In top right (above the graph of the alarm's status) is a "View in Metrics" button. Click that to go to the specific Metric that triggered the Alarm. Above the graph of the Metric, is an "Actions" button, the last option under "Actions" is "View Logs", and you can select the metric name. The next view will show all instances of the Metric and the logstream that generated the metric.
Related