How to monitor the cloud function error rate with Google Cloud Monitoring?

Question

I'd like to get an alert when a cloud function returns non-OK status codes for more than 90% of executions over 30 minutes.

The series I'm filtering is cloudfunctions.googleapis.com/function/execution_count. Out of all records, I want to count those that satisfy metric.status != 'ok'. If 90% of all records over the last 30min satisfy this condition, I want to raise an alert.

I had a look at Google Cloud's Monitoring Query Language documentation and found this part about ratios: https://cloud.google.com/monitoring/mql/examples#qlx-ratio-ratio

Here's how I wrote my condition:

fetch cloud_function
| metric 'cloudfunctions.googleapis.com/function/execution_count'
| { filter metric.status != 'ok'
  ; ident }
| group_by [resource.function_name]
| ratio
| window 30m
| condition ratio >= 0.9 '1'

It seems to work, and the graphs indicate correct values. However, it seems that as soon as a function returns an error code, the alert is raised even if there are other successful executions along compensating for the problem.

For example, here's the graph of the metrics at the time an alert was raised, which I got in Monitoring > Alerting > (click on the last alert in the events section)

The function reported in the alert is processPurchase

Is there something I'm missing that could cause the problem? Looking at the graphs, it seems the alert should not trigger. Yet, I have events and notifications about the alert. It resolves a few minutes later.

user_14004844 · Answer

I don’t think ratio could be the best option for you, due to ratio computes the ratio of value columns of two aligned input tables.

I think you could try with the following code:

fetch cloud_function
| metric 'cloudfunctions.googleapis.com/function/execution_count'
| align rate(30m)
| every 30m
| group_by [metric.status],
    [value_execution_count_percentile: percentile(value.execution_count, 90)]

As you can see I’m using the same metric cloudfunctions.googleapis.com/function/execution_count But using a Period of 30 min, and a percentage of 90% instead of the condition ratio.

Edit1

I have create a simple Cloud Function to generate random errors,

import random
def hello_world(request):
    request_json = request.get_json()
    x = random.randint(1,2)
    if x%2 ==0:
        raise NameError("my error x.x")
    else:
        return f'Hello World!'

and then I used the metric you posted as base to generate my metric:

fetch cloud_function
| metric 'cloudfunctions.googleapis.com/function/execution_count'
| { t_0:
      filter metric.status != 'ok'
      | align delta()
      | group_by [resource.function_name],
          [value_execution_count_aggregate: aggregate(value.execution_count)]
  ; t_1:
      ident
      | align delta()
      | group_by [resource.function_name],
          [value_execution_count_aggregate: aggregate(value.execution_count)] }
| ratio
| window 5m
| condition ratio >= 0.5 '1'

I only moved the ratio to ratio >= 0.5 '1' instead of 0.9 and reduced the window to 5m in order to not spend all the day getting the data.

But in my Alert Configuration I used: Condition triggers if All time series violate Edit threshold by using the condition operator in the query editor = 5

With this configuration I’m receiving the alert only when the conditions meet, and not as soon as a function returns an error code.

How to monitor the cloud function error rate with Google Cloud Monitoring?

1 Answers

Edit1