Is there any way to bind K3s / flannel to another interface?

user_14664042 0
Viewed 5010

I have a K3s (v1.20.4+k3s1) cluster with 3 nodes, each with two interfaces. The default interface has a public IP, the second one a 10.190.1.0 address. I installed K3s with and without the -flannel-backend=none option and then deployed flannel via " kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml", previously binding the kube-flannel container to the internal interface via the args "--iface=". In this setup the kube-flannel pods get the node-ip of the internal interface, but I can't reach the pods on the other nodes via ICPM. If I deploy flannel without -iface arg, the kube-flannel pods get an address from the 10.42.0.0 network. Then I can reach the pods of the other hosts, but the traffic will be routed through the public interfaces, which I want to avoid. Does anyone have a tip for me?

flannel k3s kubernetes networking interface
1 Answers

The problem was resolved in the comments section but for better visibility I decided to provide an answer.

As we can see in the K3s documentation, K3s uses flannel as the CNI by default:

By default, K3s will run with flannel as the CNI, using VXLAN as the default backend. To change the CNI, refer to the section on configuring a custom CNI.

By default, flannel selects the first interface on a host (look at the flannel documentation), but we can override this behavior with the --flannel-iface flag.
Additionally we can explicitly set IP address to advertise for node using the --node-ip flag.

I've created a simple example to illustrate how it works.

On my host machine I have two network interfaces (ens4 and ens5):

kmaster:~# ip a s | grep -i "UP\|inet"
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 127.0.0.1/8 scope host lo
2: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1460 qdisc pfifo_fast state UP group default qlen 1000
    inet 10.156.15.197/32 brd 10.156.15.197 scope global dynamic ens4
3: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1460 qdisc pfifo_fast state UP group default qlen 1000
    inet 192.168.0.2/32 brd 192.168.0.2 scope global dynamic ens5

Without setting the --flannel-iface and --node-ip flags, flannel will select the first interface (ens4: 10.156.15.197):

kmaster:~# curl -sfL https://get.k3s.io |  sh -
[INFO]  Finding release for channel stable
[INFO]  Using v1.20.4+k3s1 as release
...
[INFO]  systemd: Starting k3s
kmaster:~# kubectl get nodes -o wide  
NAME      STATUS   ROLES                  AGE   VERSION        INTERNAL-IP     
kmaster   Ready    control-plane,master   97s   v1.20.4+k3s1   10.156.15.197

But as I mentioned before we are able to override default flannel interface with the --flannel-iface flag:

kmaster:~# curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--node-ip=192.168.0.2 --flannel-iface=ens5" sh -
[INFO]  Finding release for channel stable
[INFO]  Using v1.20.4+k3s1 as release
...
[INFO]  systemd: Starting k3s
kmaster:~# kubectl get nodes -o wide
NAME      STATUS   ROLES                  AGE   VERSION        INTERNAL-IP   
kmaster   Ready    control-plane,master   64s   v1.20.4+k3s1   192.168.0.2
user_148012250
Related