no metrics returned from custom metrics API

Question

I am facing issues with HPA along with Prometheus-adapter as I am trying to push my application custom scaling metrics and based on those metrics wanted to scale Kubernetes pods up.

I am able to discover my custom metrics from custom metrics API as you can see below.(this list of metrics in output also is an intermittent.)

$kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1|jq 
{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "custom.metrics.k8s.io/v1beta1",
  "resources": [
    {
      "name": "namespaces/http_requests_per_second",
      "singularName": "",
      "namespaced": false,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    },
    {
      "name": "pods/http_requests_per_second",
      "singularName": "",
      "namespaced": true,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    },
    {
      "name": "jobs.batch/http_requests_per_second",
      "singularName": "",
      "namespaced": true,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    }
  ]
}

But having trouble with HPA as HPA is complaining about

Warning  FailedGetPodsMetric  4m57s (x128 over 69m)  horizontal-pod-autoscaler  unable to get metric http_requests_per_second: no metrics returned from custom metrics API

Here is my HPA yaml

---
kind: HorizontalPodAutoscaler
apiVersion: autoscaling/v2beta2
metadata:
  name: app1
  namespace: default
  labels:
    app.kubernetes.io/name: app1
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: app1
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: Pods
    pods:
      metric:
        name: http_requests_per_second
      target:
        type: AverageValue
        averageValue: 10m
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 80

Checked prometheus-adapter logs with log level of 7 but did not see any error, warning or any suspicious events, any thoughts what could be the problem ? Thanks in advance!

no metrics returned from custom metrics API

0 Answers