From my last question, a colleague has set up an external data source; every hour, at the bottom of the hour, Splunk calls a stored procedure in a database and gets a bit of JSON back. It looks like this:
{"counts": {
"visible":19608,"invisible":22360,"total":41968,
"breakdown_list":{
"sub_1":{ "name":"First Sub Topic" ,"total":247 ,"visible":238 ,"invisible":9 },
"sub_2":{ "name":"Second Sub Topic" ,"total":16 ,"visible":0 ,"invisible":16 },
...}
}
}
And I have my query that compares the visible count from the most-current entry with the one from 24 hours ago:
index="my_index" source="My_Dev_Stats"
| head 1
| spath path=counts.visible output=vis_now
| table vis_now host
| join host
[| search "my_index" source="My_Dev_Stats" latest=-1d
| head 1
| spath path=counts.visible output=vis_yesterday
| table vis_yesterday host ]
| eval delta=(vis_yesterday - vis_now)/vis_yesterday
I run the query, and I see the results I expect:
| vis_now | delta | vis_yesterday |
|---|---|---|
| 42371 | -0.00044862108046845483 | 42352 |
I want to know when the number of visible items drops by 10% in 24 hours, so I save this as a scheduled alert to run at 45 past the hour, and to trigger when delta > 0.10, and send me an email when it triggers.
Once I'm confident it's working correctly, the alert will create an incident ticket which will be routed to our on-call group.
But I don't get what I expect.
Graphing the number of visible items over the last week shows a monotonically-increasing curve, so the delta as calculated above is always zero or negative.
But I've gotten 12 alert emails in the last 24 hours.
Sometimes it's quiet (over night), and other times it fires every hour (I received alerts at 5:46 and 8:46 PM yesterday, and then every hour from 4:46 AM until 12:46 PM today, and then one alert at 3:46 PM).
Clicking the "View Results" link in the latest alert email shows me the query and the latest results, which have a negative value for delta, and the alert condition is delta > 0.10.
I wondered whether the timestamp skew between the current measurement and the 24-hour old measurement could be confusing the query, but I have enough data to see that the alert fired when the interval was just over 24 hours as well as just under 24 hours, and the same with the alert not firing.
Sometimes just over 24 hours, and sometimes just under.
I tried multiplying delta and the threshold by 100, so that the numbers represented percentage drops (the threshold was then delta > 10), but that didn't change the alert firing intermittently when the data says it shouldn't.