Commonly getting DNS failure calling tf.io.gfile methods from GKE: "Couldn't resolve host 'www.googleapis.com'"

user_4705168 0
Viewed 156

I have a cluster set up in Google Kubernetes Engine (GKE), with preemptible instances, TPU support, and 1 container per node.

About twice per container per day I get this error calling e.g. tf.io.gfile.glob(...):

tensorflow.python.framework.errors_impl.FailedPreconditionError: Error executing an HTTP request: libcurl code 6 meaning 'Couldn't resolve host name', error details: Couldn't resolve host 'www.googleapis.com' when reading gs://bucket/dir/dir

It only happens in GKE, not when running directly on a compute VM in Google Cloud.

Is this something I should just expect in GKE and handle (e.g. back off and retry), or is there some kind of networking config issue or other underlying problem that I can do something about?

google-kubernetes-engine tensorflow kubernetes python
1 Answers

Since your cluster has only preemptible instances, the system pods will also restart every 24 hours (at least), including kube-dns.
You have a couple of options:

  1. Create another node pool (Of-course, this node pool doesn't have to include TPU nodes, you can try and use n1-standard-1, or even cheaper instances) in that GKE cluster to host the critical pods (e.g. kube-dns), and add a nodeSelector field with that node pool to the critical deployments. For example:
nodeSelector:
        cloud.google.com/gke-nodepool: critical-node-pool
  1. The number of kube-dns replicas is determined by kube-dns-autoscaler deployment. Increasing the number of kube-dns pods can help you make your system more resilient to node preemption. You can increase by editing the kube-dns-autoscaler ConfigMap in kube-system namespace.
    Edit the ConfigMap with kubectl edit configmap dns-autoscaler --namespace=kube-system. Look for this line: linear: '{"coresPerReplica":256,"min":1,"nodesPerReplica":16}'.
    As explained here, the "min" field indicates the minimal number of DNS bakends. You can increase the minimum number of replicas, or tweak the other parameters the satisfy your needs.
    If you use the Auto-Scaling, don't set the minimum number of kube-dns replicas to an higher number than your minimal node count in that cluster, as it'll keep the nodes up just to due to deployment of kube-dns, and with no relation to your application workload.
user_124247170
Related