kube service domain name not working, but clusterIP does work with Jupyter Enterprise Gateway

user_3188040 0
Viewed 81

I have a Jupyter notebook setup in the jupyter namespace on a kubernetes cluster, and Jupyter Enterprise Gateway setup in the enterprise-gateway namespace as a Service in the same cluster.

If I configure the notebook to connect to the enterprise-gateway service using the clusterIP it works fine.

--gateway-url=http://172.20.186.249:8888

but if I switch to using the service domain name the notebook receives a 503 Connection Refused error

--gateway-url=http://enterprise-gateway.enterprise-gateway.svc.cluster.local:8888

When I use busybox check to check the kubernetes dns, the domain resolves as expected.

kubectl -n default exec -ti busybox nslookup enterprise-gateway.enterprise-gateway
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
Server:    172.20.0.10
Address 1: 172.20.0.10 kube-dns.kube-system.svc.cluster.local

Name:      enterprise-gateway.enterprise-gateway
Address 1: 172.20.186.249 enterprise-gateway.enterprise-gateway.svc.cluster.local

How do I get the domain name to work?

The Service config for the JEG looks like this...

kubectl describe svc enterprise-gateway --namespace enterprise-gateway
Name:                     enterprise-gateway
Namespace:                enterprise-gateway
Labels:                   app=enterprise-gateway
                          app.kubernetes.io/managed-by=Helm
                          chart=enterprise-gateway-2.6.0
                          component=enterprise-gateway
                          heritage=Helm
                          release=enterprise-gateway
Annotations:              meta.helm.sh/release-name: enterprise-gateway
                          meta.helm.sh/release-namespace: enterprise-gateway
Selector:                 app=enterprise-gateway
Type:                     NodePort
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       172.20.250.15
IPs:                      172.20.250.15
Port:                     http  8888/TCP
TargetPort:               8888/TCP
NodePort:                 http  31366/TCP
Endpoints:                10.1.16.136:8888,10.1.2.228:8888,10.1.30.90:8888
Port:                     response  8877/TCP
TargetPort:               8877/TCP
NodePort:                 response  31201/TCP
Endpoints:                10.1.16.136:8877,10.1.2.228:8877,10.1.30.90:8877
Session Affinity:         ClientIP
External Traffic Policy:  Cluster
Events:                   <none>
jupyter-kernel jupyter jupyter-notebook kubernetes
1 Answers

Ok, i dont know where to start i have a bunch of findings. I will start with the eye catcher one, i have a working test project i can share later on and i have to elaborate more in this answer if needed.

Step1

1- I see a mismatch on your IPs. The DNS lookup did not resolved the service DNS to the correct IP. Address 1: 172.20.186.249 is different than IP: 172.20.250.15

To debug DNS:

  1. kubectl exec "YOURPODNAME" cat /etc/resolv.conf

  2. Verify that a search path and a name server are set up correctly

    nameserver 10.96.0.10 search default.svc.cluster.local svc.cluster.local cluster.local options ndots:5

  3. check if the kubedns/coredns pods are running

    kubectl get pods --namespace=kube-system NAME READY STATUS RESTARTS AGE .... kube-dns-86f4d74b45-2qkfd 3/3 Running 232 133d kube-proxy-b2frq 1/1 Running 0 15m ...

If the pod is running, there might be something wrong with the global DNS service

kubectl get svc --namespace=kube-system
NAME      TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)     AGE kube-dns ClusterIP   10.96.0.10       <none>   53/UDP,53/TCP

You might also need to check whether DNS endpoints are exposed:

kubectl get ep kube-dns --namespace=kube-system
NAME       ENDPOINTS                     AGE
kube-dns   172.17.0.5:53,172.17.0.5:53   133d

These debugging actions will usually indicate the problem with your DNS configuration, or it will simply show you that a DNS add-on should be enabled in your cluster configuration.

Step 2

When using busybox to check the kubernetes dns This seems incorrect when looking Address 1: 172.20.186.249 im expecting to get an IP 10.X.X.X Install dnsutils on the pod as pointed below

  1. kubectl exec --stdin --tty "YOURPODNAME" -- apt update && sudo apt-get -y install dnsutils

  2. kubectl exec -it "YOURPODNAME" -- /bin/bash

  3. Inside the pod and again (weird) run apt-get install dnsutils

  4. Stay inside the pod and run nslookup "YOURSERVICENAME" you will get an IP and a Name(DNS). Check this IP since it needs to match with the IP of the service description.

  5. kubectl describe svc "YOURSERVICENAME", the IP should be the same as #4

  6. What you must see:

Ip Matching, Dns resolving

Step 3

  1. Once you have Step #2 solved you will be able to use the service name(FQDN) returned in Step 2 item #4

To be continued...

user_135797140
Related