istio-operator image is no longer available in the public istio-release registry on GCR

user_13983841 0
Viewed 581

We have recently come across an issue on one of our cluster pods, which caused an outage on our application and impacted our customers.

Here is the thing: We were able to pull the gke.gcr.io/istio/operator:1.6.3 image from GCR, though, it started failing overnight. Finally, we noticed that this image is no longer available in the public istio-release registry, on gcr.io, causing a ImagePullBackoff failure. However, we are still able to find it on docker.io.

Having said that, we're sticking with the solution approach of pulling the image from docker.io/istio/operator:1.6.3, which is a pretty straightforward one for now. Nevertheless, we're still skeptical and wondering why this image has suddenly vanished from gcr.io.

Has anyone been facing something similar?

Best regards.

google-container-registry docker-registry google-kubernetes-engine istio google-cloud-platform
2 Answers

I did some reasearch but I can't find anything related.

As I mentioned in comments, I strongly suggest you keep all critical images in a private container registry. Using this approach you can avoid incidents like that, and earn some extra control upon the images, such as: versioning, the security etc.

There are many guides on the internet to setup your own managed private container registry like Nexus, if you want to use as a service, you can try Gooogle Container Registry.

Keep in mind that when you are working in a critical environment, you need to try minize the variables to keep your service as resilient as possible.

user_119019580

I noticed a small downtime with one of our services deployed to the GKE and noticed istio-operator was listed with a red warning.

The log was:

Back-off pulling image "gke.gcr.io/istio/operator:1.6.4": ImagePullBackOff

Since istio-operator is a workload GKE manages I was hesitant but the downtime repeated couple of times for couple of minutes so I also edited the service yaml and update the image with docker.

user_13238950
Related