EKS Random "Error: ErrImagePull" "i/o timeout" when pulling Images

user_9706003 0
Viewed 1022

Running AWS "Managed Nodes" for an EKS Cluster across 2 AZ's.

3 Nodes in total. I get random timeouts when attempting to pull the containers down.

This has been so hard to trace because it does work (sometimes), so it's not like an ACL is blocking or a security group.

When I ssh into the nodes, sometimes I can pull down the image manually and sometimes I cannot. When I've run curl requests curl -I https://hub.docker.com it takes sometimes 2 minutes to get a response back. I'm guessing this is why the images are timing out.

I don't know of a way to increase the timeout for k8s to pull the image, but also can't figure out why the latency is so bad in doing the curl request.

Any suggestions are greatly appreciated.

FYI, worker nodes in Private Subnet, proper routes to NAT Gateway in place. VPC Flow logs are good.

amazon-eks kubernetes amazon-web-services
1 Answers

Random is the hardest thing to trace .

You could move your images to a private ECR registry or simply run a registry in your cluster to discard that it's an issue with your Kubernetes networking. Running AWS CNI❓

It could also just be rate-limiting from docker hub itself. Are you using the same external NAT IP address to pull from multiple nodes/clusters❓:

Docker will gradually impose download rate limits with an eventual limit of 300 downloads per six hours for anonymous users. Logged in users will not be affected at this time. Therefore, we recommend that you log into Docker Hub as an authenticated user. For more information, see the following section How do I authenticate pull requests.

✌️

user_29892610
Related