AWS Cloudfront error? invalidation max times while deploying?

I experienced a similar issue when creating multiple invalidations while deploying the same application to many distributions. The error still happens randomly, sometimes just retrying the deployment later works fine, and others fail on a different distribution.

I found a related issue showing the same error in the github aws-cdk repo, there seems to be an error with the AWS Cloudfront API during peak hours.

From the CloudFront team:

CreateInvalidation API suffers from high fault rate during the daily traffic peaks. It will return faults upto 50% of the requests. It is primarily due to the limited capacity of the API.

We have evidence that some requests failed even after six retries , during the peak. We are working on improving this , but there is no quick fix for this and we are expecting it will get better by the end of Q1 2022.

I still didn´t find a "clean" solution to this, what I have done is handle the error within my CLI script and automatically retry until it eventually succeeds.

Rather than increasing retry attempts, maybe a better idea is to introduce backoff/retry code into your deployment.

For example, when receiving the above error backoff and wait five minutes before attempting the invalidation again.

And given the number of people who configure cron jobs to run exactly on the hour, 30 minutes, etc., maybe chose a random minute value to run?

You can use increase the number of retries directly in your command:

AWS_MAX_ATTEMPTS=10 aws cloudfront create-invalidation --distribution-id=EVB1B2B3B4B56 --paths "/*"

You usually get this error when CloudFront API is experiencing a traffic peak. No guarantee to hit successfully the endpoint but you still have less chance to miss all the retries.