Kubernetes HPA on AKS is failing with error 'missing request for cpu'

user_3939022 0
Viewed 13027

I am trying to setup HPA for my AKS cluster. Following is the Kubernetes manifest file:

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    kompose.cmd: XXXXXX\tools\kompose.exe
      convert
    kompose.version: 1.21.0 (992df58d8)
  creationTimestamp: null
  labels:
    io.kompose.service: loginservicedapr
  name: loginservicedapr
spec:
  replicas: 1
  selector:
    matchLabels:
      io.kompose.service: loginservicedapr
  strategy: {}
  template:
    metadata:
      annotations:
        kompose.cmd: XXXXXX\kompose.exe
          convert
        kompose.version: 1.21.0 (992df58d8)
      creationTimestamp: null
      labels:
        io.kompose.service: loginservicedapr
    spec:
      containers:          
        image: XXXXXXX.azurecr.io/loginservicedapr:latest
        imagePullPolicy: ""
        name: loginservicedapr
        resources:
          requests:
            cpu: 250m
          limits:
            cpu: 500m            
        ports:
        - containerPort: 80
        resources: {}
      restartPolicy: Always
      serviceAccountName: ""
      volumes: null
status: {}
---
apiVersion: v1
kind: Service
metadata:
  annotations:
    kompose.cmd: XXXXXXXXXX\kompose.exe
      convert
    kompose.version: 1.21.0 (992df58d8)
  creationTimestamp: null
  labels:
    io.kompose.service: loginservicedapr
  name: loginservicedapr
spec:
  type: LoadBalancer
  ports:
  - name: "5016"
    port: 5016
    targetPort: 80
  selector:
    io.kompose.service: loginservicedapr
status:
  loadBalancer: {}

Following is my HPA yaml file:

apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:  
  name: loginservicedapr-hpa
spec:
  maxReplicas: 10 # define max replica count
  minReplicas: 3  # define min replica count
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: loginservicedapr
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 50
  - type: Pods
    pods:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 50

But when HPA is failing with the error 'FailedGetResourceMetric' - 'missing request for CPU'.

I have also installed metrics-server (though not sure whether that was required or not) using the following statement: kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.3.6/components.yaml

But still I am getting the following output when I do 'kubectl describe hpa':

Name:                                                  loginservicedapr-hpa
Namespace:                                             default
Labels:                                                fluxcd.io/sync-gc-mark=sha256.Y6dHhIOs-hNYbDmJ25Ijw1YsJ_8f0PH3Vlruj5rfbFk
Annotations:                                           fluxcd.io/sync-checksum: d5c0d9eda6db0c40f1e5e23e1356d0268dbccc8f
                                                       kubectl.kubernetes.io/last-applied-configuration:
                                                         {"apiVersion":"autoscaling/v1","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{"fluxcd.io/sync-checksum":"d5c0d9eda6db0c40f1e5...
CreationTimestamp:                                     Wed, 08 Jul 2020 17:19:47 +0530
Reference:                                             Deployment/loginservicedapr
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  <unknown> / 50%
Min replicas:                                          3
Max replicas:                                          10
Deployment pods:                                       3 current / 3 desired
Conditions:
  Type           Status  Reason                   Message
  ----           ------  ------                   -------
  AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetResourceMetric  the HPA was unable to compute the replica count: missing request for cpu
Events:
  Type     Reason                        Age                      From                       Message
  ----     ------                        ----                     ----                       -------
  Warning  FailedComputeMetricsReplicas  33m (x1234 over 6h3m)    horizontal-pod-autoscaler  Invalid metrics (1 invalid out of 1), last error was: failed to get cpu utilization: missing request for cpu
  Warning  FailedGetResourceMetric       3m11s (x1340 over 6h3m)  horizontal-pod-autoscaler  missing request for cpu

I have 2 more services that I have deployed along with 'loginservicedapr'. But I have not written HPA for those services. But I have included resource limits for those services as well in their YAML files. How to make this HPA work?

horizontalpodautoscaler azure-aks kubernetes
3 Answers

resources appears twice in your pod spec.

        resources:         # once here
          requests:
            cpu: 250m
          limits:
            cpu: 500m            
        ports:
        - containerPort: 80
        resources: {}      # another here, clearing it
user_35433710

I was able to resolve the issue by changing the following in my kubernetes manifest file from this:

resources:
          requests:
            cpu: 250m
          limits:
            cpu: 500m

to the following:

resources:
          requests:
            cpu: "250m"
          limits:
            cpu: "500m"

HPA worked after that. Following is the GitHub link which gave the solution: https://github.com/kubernetes-sigs/metrics-server/issues/237 But I did not add any Internal IP address command or anything else.

user_39390220

This is typically related to the metrics server.

Make sure you are not seeing anything unusual about the metrics server installation:

# This should show you metrics (they come from the metrics server)
$ kubectl top pods
$ kubectl top nodes

or check the logs:

$ kubectl logs <metrics-server-pod>

Also, check your kube-controller-manager logs for HPA events related entries.

Furthermore, if you'd like to explore more on whether your pods have missing requests/limits you can simply see the full output of your running pod managed by the HPA:

$ kubectl get pod <pod-name> -o=yaml

Some other people have had luck deleting and renaming the HPA too.

user_29892610
Related