In my Grafana dashboard, I count the number of "error logs" and send alert if there is at least 1 error.
This is the query I use
count_over_time(({logtype="app_log"} |= "ERROR")[1h])
This line counts the lines that includes "ERROR" in the last hour and returns the count. Then, I would like to send an alert to the dev channel, if there is such log (either 1 log, or 1000 log).
In Alert tab, I have added a rule:
Rule
- Name: ${RuleName}
- Evaluate every: 60m
- For: 0
Conditions
WHEN count () OF query (A, 60m, now) IS ABOVE 0
If no data or all values are null
SET STATE TO <OK>
If execution error or timeout
SET STATE TO <ALERTING>
I am missing some parts in here and I need help:
- I want to receive a notification at each hour, if there is an error log. Currently, this configuration wait for 1 hour, then executes the query, finds out that it has an error log, and sends notification. That is correct. Then, in the next hour, it once again searches for the logs in the last hour, finds out at least another error log, but does not send a notification, because it is already in alerting state. I want to send another notification to the dev channel because in the last execution, there is another error and I want to know it.
- The second missing part happens like in the following: In the 1st hour, there is at least 1 error and I receive a notification. Then, in the second hour there is no error and I receive again notification, but this time it is "OK". I tried different combinations but couldn't find a workaround to this. I don't want to receive a notification if there is error in the 1st hour but not in the 2nd hour.
- In the notification part, I want to know when this error happened, what was the error message, etc. I know it is doable via templating or via ${}, but in the message part, I do not have access to the log line. How can I add the erroneous log to the notification?