kubernetes Autoscaler - Cannot obtain loadbalancing.googleapis.com|https|request_count

It seems the metric that you're defining isn't available in the External Metrics API. To find out what's going on, you can inspect the External Metrics API directly:

kubectl get --raw="/apis/external.metrics.k8s.io/v1beta1" | jq

Is the loadbalancing.googleapis.com|https|request_count metric reported in the output?

You can then dig deeper by making requests of the following form:

kubectl get --raw="/apis/external.metrics.k8s.io/v1beta1/namespaces/<namespace_name>/<metric_name>?labelSelector=<selector>" | jq

And see what's returned given your metric name and a specific metric selector.

These are precisely the requests that the Horizontal Pod Autoscaler also makes at runtime. By replicating them manually, you should be able to pinpoint the source of the problem.

Comments about additional information:

1) 83m is the Kubernetes way of writing 0.083 (read as 83 "milli-units").

2) In your HorizontalPodAutoscaler definition, you use a targetAverageValue. So, if there exist multiple targets with this metric, the HPA calculates their average. So, 83m might be an average of multiple targets. To make sure, you use only the metric of a single target, you can use the targetValue field (see API reference).

3) Not sure why the items: [] array in the API response is empty. The documentation mentions that after sampling, the data is not visible for 210 seconds... You could try making the API request when the HPA is not running.

Thank you very much for your detailed response.

When using the metricSelector to select the specific forwarding_rule_name, we need to use the exact forwarding_rule_name as defined by the ingress:

metricSelector:
    matchLabels:
    resource.labels.forwarding_rule_name: k8s-fws-default-serviceb--3a908157de956ba7

$ kubectl describe ingress

Name: serviceb
...

Annotations:
  ingress.kubernetes.io/https-forwarding-rule:  k8s-fws-default-serviceb--9bfb478c0886702d
  ...
  kubernetes.io/ingress.allow-http:             false
  kubernetes.io/ingress.global-static-ip-name:  static-ip

The problem, is that the suffix of the forwarding_rule_name (3a908157de956ba7) changes for every deployment, and is created dynamically on Ingress creation:

• k8s-fws-default-serviceb--3a908157de956ba7

We have a fully automated deployment using Helm, and, as such, when the HPA is created, we don't know what the forwarding_rule_name will be.

And, it seems that the matchLabels does not accept regular expressions, or else we would simply do something like:

metricSelector:
    matchLabels:
    resource.labels.forwarding_rule_name: k8s-fws-default-serviceb--*

I've tried several approaches, all without success:

1. Use Annotations to force the forwarding_rule_name
2. Use a different machLabel, as backend_target_name
3. Obtain the forwarding_rule_name using a command, so i can insert it later in the yaml file.

Use Annotations to force the forwarding_rule_name:

When creating the ingress, i can use specific annotations to change the default behavior, or define specific values, for example, on Ingress.yaml:

  annotations:
    kubernetes.io/ingress.global-static-ip-name: static-ip

I tried to use the https-forwarding-rule annotation to force a specific "static" name, but this didn't work:

  annotations:
    ingress.kubernetes.io/https-forwarding-rule: some_name


  annotations:
    kubernetes.io/https-forwarding-rule: some_name

Use a different machLabel, as backend_target_name

metricSelector:
        matchLabels:
          resource.labels.backend_target_name: serviceb

Also failed.

Obtain the forwarding_rule_name using a command

When executing the following command, i get the list of Forwarding Rules, but for all the clusters. And according to the documentation, is not possible to filter by cluster:

gcloud compute forwarding-rules list

NAME                                        P_ADDRESS   IP_PROTOCOL  TARGET
k8s-fws-default-serviceb--4e1c268b39df8462  xx          TCP          k8s-tps-default-serviceb--4e1c268b39df8462
k8s-fws-default-serviceb--9bfb478c0886702d  xx          TCP          k8s-tps-default-serviceb--9bfb478c0886702d

Is there any way to allow me to select the resource i need, in order to get the Requests count metric?

It seems everything was OK with my code, but, there is a time delay (aprox. 10m), before the request_count metric is available. After this period, the metric is now computed and available:

$ kubectl get hpa ServiceB
NAME                REFERENCE    TARGETS                MINPODS   MAXPODS   REPLICAS   AGE
ServiceB   Deployment/ServiceB   83m/100 (avg), 1%/80%  1         3         1          18m

Now, regarding the loadbalancing.googleapis.com|https|request_count metric, i'm not understanding how its being presented. What does 83m means?

According to Google documentation for Load balancing metrics:

https/request_bytes_count Request bytes

DELTA, INT64, By

GA

The number of requests served by HTTP/S load balancer. Sampled every 60 seconds. After sampling, data is not visible for up to 210 seconds.

According to Metric Details:

In a DELTA metric, each data point represents the change in a value over the time interval. For example, the number of service requests received since the previous measurement would be a delta metric.

I've made one single request to the service, so i was expecting a value of 1, and i can't understand what the 83m means.

Another possibility, could be that i'm not using the correct metric. I've selected the loadbalancing.googleapis.com|https|request_count metric, assuming it would provide the number of requests that were executed by the service, via the loadbalancer.

Isn't exactly this information that the loadbalancing.googleapis.com|https|request_count metric provides?

Regarding the above comment, when executing:

kubectl get --raw="/apis/external.metrics.k8s.io/v1beta1/namespaces/default/pubsub.googleapis.com|subscription|num_undelivered_messages" | jq

i get the correct data:

... { "metricName": "pubsub.googleapis.com|subscription|num_undelivered_messages", "metricLabels": { "resource.labels.project_id": "project-id", "resource.labels.subscription_id": "subscription_id", "resource.type": "pubsub_subscription" }, "timestamp": "2019-10-22T15:39:58Z", "value": "4" } ...

but, when executing:

kubectl get --raw="/apis/external.metrics.k8s.io/v1beta1/namespaces/default/loadbalancing.googleapis.com|https|request_count" | jq

i get nothing back:

{ "kind": "ExternalMetricValueList", "apiVersion": "external.metrics.k8s.io/v1beta1", "metadata": { "selfLink": >"/apis/external.metrics.k8s.io/v1beta1/namespaces/default/loadbalancing.googleapis.com%7Chttps%7Crequest_count" }, "items": [] }