Celery HPA - startupProbe failing after tasks are already running

Question

I have Celery workers running on Kubernetes 1.20 cluster on AWS EKS using AWS Elasticache Redis as the broker. Because of the nature of the project ~80% of the time celery workers are running idle so the logical thing was to have them scale automatically. Scaling based on CPU/memory works ok. At about 4 workers node scaling also needs to kick in and that works ok as well. An obvious problem is that it takes some time for a new node to start and get fully operational before that node can start taking on new celery worker pods. So some waiting to scale up is expected.

Somewhere in all that waiting a fresh celery worker pod gets started and starts accepting new tasks and executing them, but for some unknown reason the startupProbe does not complete. Because startupProbe was not successful the whole pod is killed potentially in the middle of a running task.

Question: Can I prevent celery from taking on tasks before the startupProbe is considered successful?

Celery config

apiVersion: apps/v1
kind: Deployment
metadata:
  name: celery-worker
  labels:
    app: celery-worker
spec:
  selector:
    matchLabels:
      app: celery-worker
  progressDeadlineSeconds: 900
  template:
    metadata:
      labels:
        app: celery-worker
    spec:
      containers:
      - name: celery-worker
        image: -redacted-
        imagePullPolicy: Always
        command: ["./scripts/celery_worker_entrypoint_infra.sh"]
        env:
          - name: CELERY_BROKER_URL
            valueFrom:
              secretKeyRef:
                name: celery-broker-url-secret
                key: broker-url
        startupProbe:
          exec:
            command: ["/bin/bash", "-c", "celery -q -A app inspect -d celery@$HOSTNAME --timeout 10 ping"]
          initialDelaySeconds: 20
          timeoutSeconds: 10
          successThreshold: 1
          failureThreshold: 30
          periodSeconds: 10
        readinessProbe:
          exec:
            command: ["/bin/bash", "-c", "celery -q -b $CELERY_BROKER_URL inspect -d celery@$HOSTNAME --timeout 10 ping"]
          periodSeconds: 120
          timeoutSeconds: 10
          successThreshold: 1
          failureThreshold: 3
        livenessProbe:
          exec:
            command: ["/bin/bash", "-c", "celery -q -b $CELERY_BROKER_URL inspect -d celery@$HOSTNAME --timeout 10 ping"]
          periodSeconds: 120
          timeoutSeconds: 10
          successThreshold: 1
          failureThreshold: 5
        resources:
          requests:
            memory: "384Mi"
            cpu: "250m"
          limits:
            memory: "1Gi"
            cpu: "500m"
      terminationGracePeriodSeconds: 2400

Celery HPA config

kind: HorizontalPodAutoscaler
apiVersion: autoscaling/v2beta2
metadata:
  name: celery-worker
spec:
  minReplicas: 2
  maxReplicas: 40
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: celery-worker
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 60
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 90

Celery startup script

python manage.py check
exec celery --quiet -A app worker \
        --loglevel info \
        --concurrency 1 \
        --uid=nobody \
        --gid=nogroup

I'm sharing complete configs including readinessProbe and livenessProbe whose values are a bit inflated, but are a consequence of various try-and-error scenarios.

Edit: This is a catch-22 situation.

I have defined startupProbe to check if celery is running in current host and that will only be true if celery worker is running. And if celery worker is running it will accept tasks. And if it will accept tasks celery inspect command might take too long causing the startupProbe to hang and failing. If startupProbe fails too many times it will kill the pod.

Furthermore, if I call celery inspect without destination (host) defined, startupProbe will fail on initial deployment.

Conclusion: celery inspect is not a good startupProbe candidate.

user_15407542 · Answer

Conclusion: celery inspect is not a good startupProbe candidate.

I agree.

You also mentioned that you have an active healtcheck

I have in another container a web service that uses Django framework and exposes health check at /health

I think it is worth using it to create startup probe:

Sometimes, you have to deal with legacy applications that might require an additional startup time on their first initialization. In such cases, it can be tricky to set up liveness probe parameters without compromising the fast response to deadlocks that motivated such a probe. The trick is to set up a startup probe with the same command, HTTP or TCP check, with a failureThreshold * periodSeconds long enough to cover the worse case startup time.

You also mention an example attack:

Hmm, I'm not sure that is a smart thing to do. If for instance my web service that serves /health gets DDoS-ed my celery workers would also fail.

However, this shouldn't be a problem. Startup probe will only run when the container is started. The chance that someone will attack you while the environment is being launched is practically zero. You should use readiness or liveness probe to check if your container is alive while the application is running.

The kubelet uses startup probes to know when a container application has started. If such a probe is configured, it disables liveness and readiness checks until it succeeds, making sure those probes don't interfere with the application startup. This can be used to adopt liveness checks on slow starting containers, avoiding them getting killed by the kubelet before they are up and running.

Here you can find example yaml

ports:
- name: liveness-port
  containerPort: 8080
  hostPort: 8080

livenessProbe:
  httpGet:
    path: /healthz
    port: liveness-port
  failureThreshold: 1
  periodSeconds: 10

startupProbe:
  httpGet:
    path: /healthz
    port: liveness-port
  failureThreshold: 30
  periodSeconds: 10

In this case startupProbe will execute on their first initialization, then will be used livenessProbe.

Celery HPA - startupProbe failing after tasks are already running

1 Answers