Why the number of pods running is reduced while the number of messages in the queue is still high?

Viewed 85

I am running a KEDA enabled k8s cluster with Storage Queue Azure Function which executes 2000 messages at once having variable execution times (between 30 seconds to 10 minutes). The system works fine for a certain duration but when the number of messages in the queue starts decreasing, the number of pods also start decreasing.

Question: Why the number of pods running is reduced while the number of messages in the queue is still high?

Example: The maximum replica count is 120 and the number of messages in the queue is around 250. So why the HPA scales down the pods. Ideally, it should still be using all the 120 pods to complete all messages in the queue.

Supplement Questions:

  1. Does queueLength affects the behavior of HPA?
  2. Few messages were not executed fully. It seems they were shut down abruptly during the execution though terminationGracePeriodSeconds is 10 minutes. Any comments?

Following is the scaledobject.yaml file:

apiVersion: v1
kind: Secret
metadata:
  name: queue-connection-secret
data:
  connection-string: ####
---
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: azure-queue-auth
spec:
  secretTargetRef:
  - parameter: connection
    name: queue-connection-secret
    key: connection-string
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: queuetrigfuncscaledobject
spec:
  scaleTargetRef:
    name: queuetrigfuncdeployment
  minReplicaCount: 0
  maxReplicaCount: 120
  pollingInterval: 1
  cooldownPeriod: 900

  triggers:
  - type: azure-queue
    metadata:
      queueName: k8s-poc-queue
      queueLength: "1"
    authenticationRef:
        name: azure-queue-auth

Deployments.yaml file:

apiVersion : apps/v1
kind: Deployment
metadata:
  name: queuetrigfuncdeployment
  labels:
    app: queuetrigfuncpod
spec:
  selector:
    matchLabels:
      app: queuetrigfuncpod
  template:
    metadata:
      labels:
        app: queuetrigfuncpod
    spec:
      containers:
        - image: #######.azurecr.io/#######
          name: queuetrigcontainer
          ports:
          - containerPort: 80
          resources:
            requests:
              memory: "500Mi"
              cpu: "700m"
            limits:
              memory: "600Mi"
              cpu: "700m"
      nodeSelector:
        agentpool: testuserpool
      terminationGracePeriodSeconds: 600
      imagePullSecrets:
      - name: regcred
1 Answers

That scaler is using AverageValue mode so the count is getting divided by the number of pods. You want Value mode instead, but there is no option for that currently.

This is an overall issue in Keda that we keep meaning to fix globally but no one has had the time. If you put a PR, ping me in Slack and I can review it.

Related