I am running a KEDA enabled k8s cluster with Storage Queue Azure Function which executes 2000 messages at once having variable execution times (between 30 seconds to 10 minutes). The system works fine for a certain duration but when the number of messages in the queue starts decreasing, the number of pods also start decreasing.
Question: Why the number of pods running is reduced while the number of messages in the queue is still high?
Example: The maximum replica count is 120 and the number of messages in the queue is around 250. So why the HPA scales down the pods. Ideally, it should still be using all the 120 pods to complete all messages in the queue.
Supplement Questions:
- Does queueLength affects the behavior of HPA?
- Few messages were not executed fully. It seems they were shut down abruptly during the execution though terminationGracePeriodSeconds is 10 minutes. Any comments?
Following is the scaledobject.yaml file:
apiVersion: v1
kind: Secret
metadata:
name: queue-connection-secret
data:
connection-string: ####
---
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
name: azure-queue-auth
spec:
secretTargetRef:
- parameter: connection
name: queue-connection-secret
key: connection-string
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: queuetrigfuncscaledobject
spec:
scaleTargetRef:
name: queuetrigfuncdeployment
minReplicaCount: 0
maxReplicaCount: 120
pollingInterval: 1
cooldownPeriod: 900
triggers:
- type: azure-queue
metadata:
queueName: k8s-poc-queue
queueLength: "1"
authenticationRef:
name: azure-queue-auth
Deployments.yaml file:
apiVersion : apps/v1
kind: Deployment
metadata:
name: queuetrigfuncdeployment
labels:
app: queuetrigfuncpod
spec:
selector:
matchLabels:
app: queuetrigfuncpod
template:
metadata:
labels:
app: queuetrigfuncpod
spec:
containers:
- image: #######.azurecr.io/#######
name: queuetrigcontainer
ports:
- containerPort: 80
resources:
requests:
memory: "500Mi"
cpu: "700m"
limits:
memory: "600Mi"
cpu: "700m"
nodeSelector:
agentpool: testuserpool
terminationGracePeriodSeconds: 600
imagePullSecrets:
- name: regcred