progressDeadlineSeconds is not correctly timing out

user_1028270 0
Viewed 5460

I have progressDeadlineSeconds set to 120 seconds

I deploy and run kubectl rollout status deployment mydeployment

The deployment failed with 0 of 1 updated replicas available - CrashLoopBackOff

But the kubectl is still hanging forever with message: Waiting for deployment "mydeployment" rollout to finish: 0 of 1 updated replicas are available...

Why is this happening, progressDeadlineSeconds is suppose to force it to fail and cause kubectl rollout status deployment to exit with a non-zero return code right?

kubectl kubernetes
1 Answers

You are correct, kubectl rollout status returns a non-zero exit code if the Deployment has exceeded the progression deadline. Progress Deadline Seconds:

.spec.progressDeadlineSeconds is an optional field that specifies the number of seconds you want to wait for your Deployment to progress before the system reports back that the Deployment has failed progressing - surfaced as a condition with Type=Progressing, Status=False. and Reason=ProgressDeadlineExceeded in the status of the resource. The Deployment controller will keep retrying the Deployment. This defaults to 600. In the future, once automatic rollback will be implemented, the Deployment controller will roll back a Deployment as soon as it observes such a condition.

If specified, this field needs to be greater than .spec.minReadySeconds.

Which brings us to Min Ready Seconds

.spec.minReadySeconds is an optional field that specifies the minimum number of seconds for which a newly created Pod should be ready without any of its containers crashing, for it to be considered available. This defaults to 0 (the Pod will be considered available as soon as it is ready).

Without your exact deployment configs it is hard to tell where the problem is but there are few things to check out:

  • Try setting both the progressDeadlineSeconds and minReadySeconds remembering that the later must have a smaller value.

  • progressDeadlineSeconds might not be respected when used with Replicas: 1. Check your maxUnavailable param in order to see if it allows deployment full unavailability.

  • As a workaround you can specify a timeout period for your kubectl rollout status command. For example: kubectl rollout status deployment mydeployment --timeout=120s

  • If you don't want to wait for the rollout to finish then you can use --watch=false. You will than have to check the status manually by running kubectl describe deployment and kubectl get deployment commands which might not be ideal.

Please let me know if that helps.

user_115608780
Related