Setup Grafana alert on Loki's errors log

I have setup Grafana and loki (2.6.1 / 9.0), and I'd like to setup an alert whenever logs have Error as severity.

I have this query:

count (rate({namespace="api-dev"} |= `"severity":"Error"` [5m]))

Which gives me what I want:

However, even few hours later, the alert is still firing.

Is there a proper way to be notified whenever a new error log is found, but not too much (eg. not more than twice per hour)?

PS: I had a look at this answer, but the issue persists.