kubernetes kubedns - sidecar and masq crashes if I do a nslookup

user_8346516 0
Viewed 2209

I am facing a strange issue Here are my kubernetes details

Kubernetes version:  1.6.4
OS: Ubuntu 17.04

I use the YAML files from the following link to install kube-dns https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/dns/

Step 1

I then try to launch busybox using 

kubectl run -i --tty busybox --image=busybox – sh

Now when I do nslookup kubernetes it works. Now when I try to see the docker logs for my sidecar I see the following

ERROR: logging before flag.Parse: I0721 17:15:25.067730       1 main.go:48] Version v1.14.3-4-gee838f6
ERROR: logging before flag.Parse: I0721 17:15:25.067981       1 server.go:45] Starting server (options {DnsMasqPort:53 DnsMasqAddr:127.0.0.1 DnsMasqPollIntervalMs:5000 Probes:[{Label:kubedns Server:127.0.0.1:10053 Name:kubernetes.default.svc.cluster.local. Interval:5s Type:1} {Label:dnsmasq Server:127.0.0.1:53 Name:kubernetes.default.svc.cluster.local. Interval:5s Type:1}] PrometheusAddr:0.0.0.0 PrometheusPort:10054 PrometheusPath:/metrics PrometheusNamespace:kubedns})
ERROR: logging before flag.Parse: I0721 17:15:25.068191       1 dnsprobe.go:75] Starting dnsProbe {Label:kubedns Server:127.0.0.1:10053 Name:kubernetes.default.svc.cluster.local. Interval:5s Type:1}
ERROR: logging before flag.Parse: I0721 17:15:25.068398       1 dnsprobe.go:75] Starting dnsProbe {Label:dnsmasq Server:127.0.0.1:53 Name:kubernetes.default.svc.cluster.local. Interval:5s Type:1}
ERROR: logging before flag.Parse:

Step 2 If I do a nslookup against a name (say nslookup ABCD) that I know does not exist in kubedns (as a A record) then I see following logs in my dnsmesg container

I0718 17:26:33.738368       1 nanny.go:108] dnsmasq[13]: Maximum number of concurrent DNS queries reached (max: 150)

And at the same time I see following in sidecar

dns sidecar errorERROR: logging before flag.Parse: I0720 02:09:37.975054       1 main.go:48] Version v1.14.3-4-gee838f6
ERROR: logging before flag.Parse: I0720 02:09:37.975140       1 server.go:45] Starting server (options {DnsMasqPort:53 DnsMasqAddr:127.0.0.1 DnsMasqPollIntervalMs:5000 Probes:[{Label:kubedns Server:127.0.0.1:10053 Name:kubernetes.default.svc.cluster.local. Interval:5s Type:1} {Label:dnsmasq Server:127.0.0.1:53 Name:kubernetes.default.svc.cluster.local. Interval:5s Type:1}] PrometheusAddr:0.0.0.0 PrometheusPort:10054 PrometheusPath:/metrics PrometheusNamespace:kubedns})
ERROR: logging before flag.Parse: I0720 02:09:37.975160       1 dnsprobe.go:75] Starting dnsProbe {Label:kubedns Server:127.0.0.1:10053 Name:kubernetes.default.svc.cluster.local. Interval:5s Type:1}
ERROR: logging before flag.Parse: I0720 02:09:37.975200       1 dnsprobe.go:75] Starting dnsProbe {Label:dnsmasq Server:127.0.0.1:53 Name:kubernetes.default.svc.cluster.local. Interval:5s Type:1}
ERROR: logging before flag.Parse: W0720 02:12:50.090595       1 server.go:64] Error getting metrics from dnsmasq: read udp 127.0.0.1:36473->127.0.0.1:53: i/o timeout
ERROR: logging before flag.Parse: W0720 02:13:03.607740       1 server.go:64] Error getting metrics from dnsmasq: read udp 127.0.0.1:56334->127.0.0.1:53: i/o timeout
ERROR: logging before flag.Parse: W0720 02:13:10.609651       1 server.go:64] Error getting metrics from dnsmasq: read udp 127.0.0.1:39976->127.0.0.1:53: i/o timeout
ERROR: logging before flag.Parse: W0720 02:13:23.644035       1 server.go:64] Error getting metrics from dnsmasq: read udp 127.0.0.1:57226->127.0.0.1:53: i/o timeout

Step 3 Now here is a bummer after 5minutes or 10 minutes my sidecar and dnsmasq both crash and new containers get recreated.

A few other details about my K8s cluster I am using VMWare workstation 12 Pro to run my cluster on one node and I have enabled RBAC and am using client certificates for authentication

SOS. Can someone point me in the right direction? Have spent a lot of time trying to figure this one out. Thanks in advance

dns
2 Answers

I had the same problem with Ubuntu 18.04, where on top of /etc/resolve.conf and systemd-resolved a third system with netplan comes into play. But that doesn´t bother kube-dns - as already described on GitHub, kube-dns simply picks the contents of /etc/resolve.conf which was in my case: 

nameserver 127.0.0.53

It turned out, that this is simply a known Ubuntu systemd-resolved bug (see this and this so answers). Long story short, there´s only one workaround (really, I also didn´t believe it and tried to use systemd-resolved configuration editing /etc/systemd/resolved.conf and restarting the service - but that didn´t work for me). Just (re)move the original /etc/resolv.conf and replace it with a link to /run/systemd/resolve/resolv.conf:

sudo mv /etc/resolv.conf /etc/resolv.conf_orig
sudo ln -s /run/systemd/resolve/resolv.conf /etc/resolv.conf

Now your /etc/resolve.conf should have the correct nameserver, which is nameserver 10.0.2.3 in my case, where I´am using VirtualBox (that may differ for your setup!). Now after kube-dns was re-applied, it picks up the correct nameserver for every nslookup and everything works fine:

kubectl exec -it busybox-68654f944b-7rdzs -- nslookup kubernetes

If you´re interested, here´s an implementation with Ansible for that problem: https://github.com/jonashackt/kubernetes-the-ansible-way/issues/6

Update 08/29/2018: The current busybox image has DNS lookup problems (see https://github.com/kubernetes/kubernetes/issues/66924). Try to run version 1.28.4 explicitely with:

kubectl run busybox --image=busybox:1.28.4 --command -- sleep 3600

This should prevent you from days of searching!

user_49645530

I forgot to mention that I'm running Ubuntu 16.04 which AFAIK does not run systemd-resolved

Which explains why I could not find /run/systemd/resolve(resolv.conf) folder to symlink my /etc/resolv.conf to

Apparently systemd-resolved was introduced in version 16.10

I did switch to using version 1.28.4 of busybox but that didn't help any either.

user_103426370
Related