For an alert condition, Prometheus is behaving incorrectly.
The alert gets generated based on the condition and goes to different notification channels correctly. The problem is - after some time the alert resolves (at random time gaps, sometimes within 5 minutes and sometimes after 10 minutes) and then re-generates within a minute. When I check the condition it still matches for the alert to not to resolve.
In fact for a particular scenario, the alert was supposed to be active for 6 hours but during that period I received at least 15 resolved notifications and re-generation of the alert post resolution. I have checked for missing telemetry but telemetry was always present. There is no time offset in the alert condition. This does not make any sense because it is not behaving like this for any other alert and I have at least 80 alerts running in my production environment. Following is the query which is plain:
(sum by(pcName) (cm_summary{regState="p_online"}) == 0 and sum by(pcName) (cm_summary{regState="nb_online"}) == 0 and sum by(pcName) (cm_summary{regState="f40_online"}) == 0 ) and on(pcName) (sum(cm_summary) by (pcName) >= 5)