Why is HPA scale up even the usage doesn't hit the threshold?

user_5421539 0
Viewed 94

I have deployed a HPA the configuration showed at the bottom. It scales up when either CPU or Memory usage is above 75%. The initial replicas count is 1 and the max is 3. But I can see the pod count was scaled up to 3 1 few minutes after I deploy the HPA.

The current usage of CPU/Memory is shown below. You can see it is quit low compare the requested resources which is 2 CPU and 8GB memory. I don't understand why it scales. Did I make any mistake on the configuration?

$ kubectl top pod transform-67df4445c5-6qpdd
W0818 16:04:43.199730   63930 top_pod.go:140] Using json format to get metrics. Next release will switch to protocol-buffers, switch early by passing --use-protocol-buffers flag
NAME                         CPU(cores)   MEMORY(bytes)
transform-67df4445c5-6qpdd   250m         495Mi

apiVersion: apps/v1
kind: Deployment
metadata:
  name: transform
  namespace: default
spec:
  replicas: 1

  selector:
    matchLabels:
      name: transform
  template:
    metadata:
      labels:
        name: transform
    spec:
      containers:
        - name: transform
          image: zhaoyi0113/es-kinesis-firehose-transform
          resources:
            requests:
              cpu: 2
              memory: 8
          ports:
            - containerPort: 8080
---
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: transform
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: transform
  minReplicas: 1
  maxReplicas: 3
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 75
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 75
kubernetes
1 Answers

You have mentioned the resource without unit : https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#resource-units-in-kubernetes

resources:
            requests:
              cpu: 2
              memory: 8

for memory, it could be 8 Mi and usage is 450Mi so it's above. That also could be a reason. You have not mentioned the limit for resource so it always best practice to add the limit also in resource.

So that HPA can calculate the % based on request and limit you set to resource section.

You can also check the

kubectl get hpa

or

kubectl describe hpa <name>

to check the usage % and event details.

here is nice best practice article from google : https://cloud.google.com/blog/products/containers-kubernetes/kubernetes-best-practices-resource-requests-and-limits

user_55258240
Related