HPA cannot read metric value (CPU utilization) on GKE

user_9764480 0
Viewed 2248

I'm working on Google Kubernetes Engine on a single cluster. The cluster automatically scales the number of nodes. I have three created a Deployment and set up the auto-scaling policy using the website (Workloads -> Deployment -> Actions -> Auto-scaling), so not manually writing the YAML configuration. Based on an official guide, I did not make any mistake.

If you do not specify requests, you can autoscale based only on the absolute value of the resource's utilization, such as milliCPUs for CPU utilization.

The following is the full deployment YAML:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: student
  name: student
  namespace: ulibretto
spec:
  replicas: 1
  selector:
    matchLabels:
      app: student
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: student
    spec:
      containers:
        - env:
            - name: CLUSTER_HOST
              valueFrom:
                configMapKeyRef:
                  key: CLUSTER_HOST
                  name: shared-env-vars
            - name: BIND_HOST
              valueFrom:
                configMapKeyRef:
                  key: BIND_HOST
                  name: shared-env-vars
            - name: TOKEN_TIMEOUT
              valueFrom:
                configMapKeyRef:
                  key: TOKEN_TIMEOUT
                  name: shared-env-vars
          image: gcr.io/ulibretto/github.com/ulibretto/studentservice
          imagePullPolicy: IfNotPresent
          name: studentservice-1
---
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  labels:
    app: student
  name: student-hpa-n3bp
  namespace: ulibretto
spec:
  maxReplicas: 100
  metrics:
    - resource:
        name: cpu
        targetAverageUtilization: 80
      type: Resource
  minReplicas: 1
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: student
---
apiVersion: v1
kind: Service
metadata:
  annotations:
    cloud.google.com/neg: '{"ingress":true}'
  labels:
    app: student
  name: student-ingress
  namespace: ulibretto
spec:
  clusterIP: 10.44.5.59
  ports:
    - port: 5000
      protocol: TCP
      targetPort: 5000
  selector:
    app: student
  sessionAffinity: None
  type: ClusterIP

The problem is that the HPA does not see the metric (average CPU utilization), which is really strange (see image). HPA cannot read metric value

What I am missing?

google-kubernetes-engine gcloud kubernetes yaml
1 Answers

EDITED

You are right. You don't need to specify namespace: ulibretto in scaleTargetRef: as I mentioned earlier.

As you provided all YAMLs I was able to find proper root cause.

If you will check GKE docs you will find comment in code

    resources:
      # You must specify requests for CPU to autoscale
      # based on CPU utilization
      requests:
        cpu: "250m"

Your deployment didn't have specified resource requests. I've tried on this (I've removed some parts as I was not able to deploy your container and changed apiVersion in HPA):

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: student
  name: student
  namespace: ulibretto
spec:
  replicas: 3
  selector:
    matchLabels:
      app: student
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: student
    spec:
      containers:
      - image: nginx
        imagePullPolicy: IfNotPresent
        name: studentservice-1
        resources:
          requests:
            cpu: "250m"
---
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  labels:
    app: student
  name: student-hpa
  namespace: ulibretto
spec:
  maxReplicas: 100
  minReplicas: 1
  targetCPUUtilizationPercentage: 80
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: student

$ kubectl get all -n ulibretto
NAME                           READY   STATUS    RESTARTS   AGE
pod/student-6f797d5888-84xfq   1/1     Running   0          7s
pod/student-6f797d5888-b7ctq   1/1     Running   0          7s
pod/student-6f797d5888-fbtmd   1/1     Running   0          7s
NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/student   3/3     3            3           7s
NAME                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/student-6f797d5888   3         3         3       7s
NAME                                              REFERENCE            TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
horizontalpodautoscaler.autoscaling/student-hpa   Deployment/student   <unknown>/80%   1         100       0          7s

After ~1-5 minutes you will receive some metrics.

$ kubectl get all -n ulibretto
NAME                           READY   STATUS    RESTARTS   AGE
pod/student-6f797d5888-84xfq   1/1     Running   0          95s
pod/student-6f797d5888-b7ctq   1/1     Running   0          95s
pod/student-6f797d5888-fbtmd   1/1     Running   0          95s

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/student   3/3     3            3           95s

NAME                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/student-6f797d5888   3         3         3       95s

NAME                                              REFERENCE            TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
horizontalpodautoscaler.autoscaling/student-hpa   Deployment/student   0%/80%    1         100       3          95s

Same situation if you would like to create HPA using CLI:

$ kubectl autoscale deployment student -n ulibretto --cpu-percent=50 --min=1 --max=100
horizontalpodautoscaler.autoscaling/student autoscaled

$ kubectl get hpa -n ulibretto
NAME      REFERENCE            TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
student   Deployment/student   <unknown>/50%   1         100       0          3s

And after a while you will receive 0% instead of <unknown>

$ kubectl get all -n ulibretto
NAME                           READY   STATUS    RESTARTS   AGE
pod/student-6f797d5888-84xfq   1/1     Running   0          4m4s
pod/student-6f797d5888-b7ctq   1/1     Running   0          4m4s
pod/student-6f797d5888-fbtmd   1/1     Running   0          4m4s
NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/student   3/3     3            3           4m5s
NAME                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/student-6f797d5888   3         3         3       4m5s
NAME                                          REFERENCE            TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
horizontalpodautoscaler.autoscaling/student   Deployment/student   0%/50%    1         100       3          58s
user_111481390
Related