Kubectl port-forwarding not working for IPv6 binding with socat

Question

I'm trying to understand why this particular socat command isn't working in my case where I run it in a IPv6 only Kubernetes cluster.

Cluster is build on top of AWS with Calico CNI & containerd. Provisioned using kubeadm and Kubernetes 1.21.

I have run the following socat command which binds to loopback interface ::1,

kubectl --context=$CLUSTER1 run --image=alpine/socat socat -- tcp6-listen:15000,bind=$$::1$$,fork,reuseaddr /dev/null

And then I try to port-forward and curl to 15000 port,

kubectl --context=$CLUSTER1 port-forward pod/socat 35000:15000 --address=::1
curl -ivg http://localhost:35000

I get the error,

Forwarding from [::1]:35000 -> 15000
Handling connection for 35000
E0830 17:09:59.604799   79802 portforward.go:400] an error occurred forwarding 35000 -> 15000: error forwarding port 15000 to pod a8ba619774234e73f4c1b4fe4ff47193af835cffc56cb6ad1a8f91e745ac74e9, uid : failed to execute portforward in network namespace "/var/run/netns/cni-8bade2c1-28c9-6776-5326-f10d55fd0ff9": failed to dial 15000: dial tcp4 127.0.0.1:15000: connect: connection refused

Its listening to 15000 as,

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 ::1:15000               :::*                    LISTEN      1/socat

However if I run the following it works fine,

kubectl --context=$CLUSTER1 run --image=alpine/socat socat -- tcp6-listen:15000,bind=$$::$$,fork,reuseaddr /dev/null

Not sure I understand why port-forward would fail for the loopback interface binding ::1 but not for catch all ::. Can someone please shed some light on this ?

user_429199 · Answer

For those of you running into a similar issue with your IPv6 only Kubernetes clusters heres what I have investigated found so far.

Background: It seems that this is a generic issue relating to IPv6 and CRI. I was running containerd in my setup and containerd versions 1.5.0-1.5.2 added two PRs (don't use socat for port forwarding and use happy-eyeballs for port-forwarding) which fixed a number of issues in IPv6 port-forwarding.

Potential fix: Further to pulling in containerd version 1.5.2 (as part of Ubuntu 20.04 LTS) I was also getting the error IPv4: dial tcp4 127.0.0.1:15021: connect: connection refused IPv6 dial tcp6: address localhost: no suitable address found when port-forwarding. This is caused by a DNS issue when resolving localhost. Hence I added localhost to resolve as ::1 in the host machine with the following command.

sed -i 's/::1 ip6-localhost ip6-loopback/::1 localhost ip6-localhost ip6-loopback/' /etc/hosts

I think the important point here is that check your container runtimes to make sure IPv6 (tcp6 binding) is supported.

Kubectl port-forwarding not working for IPv6 binding with socat

1 Answers