How do GCP load balancers route traffic to GKE services?

Question

I'm relatively new (< 1 year) to GCP, and I'm still in the process of mapping the various services onto my existing networking mental model.

Once knowledge gap I'm struggling to fill is how HTTP requests are load balanced to services running in our GKE clusters.

On a test cluster, I created a service in front of pods that serve HTTP:

apiVersion: v1
kind: Service
metadata:
  name: contour
spec:
 ports:
 - port: 80
   name: http
   protocol: TCP
   targetPort: 8080
 - port: 443
   name: https
   protocol: TCP
   targetPort: 8443
 selector:
   app: contour
 type: LoadBalancer

The service is listening on node ports 30472 and 30816.:

$ kubectl get svc contour
NAME      TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGE
contour   LoadBalancer   10.63.241.69   35.x.y.z   80:30472/TCP,443:30816/TCP   41m

A GCP network load balancer is automatically created for me. It has its own public IP at 35.x.y.z, and is listening on ports 80-443:

Curling the load balancer IP works:

$ curl -q -v 35.x.y.z
* TCP_NODELAY set
* Connected to 35.x.y.z (35.x.y.z) port 80 (#0)
> GET / HTTP/1.1
> Host: 35.x.y.z
> User-Agent: curl/7.62.0
> Accept: */*
> 
< HTTP/1.1 404 Not Found
< date: Mon, 07 Jan 2019 05:33:44 GMT
< server: envoy
< content-length: 0
<

If I ssh into the GKE node, I can see the kube-proxy is listening on the service nodePorts (30472 and 30816) and nothing has a socket listening on ports 80 or 443:

# netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 127.0.0.1:20256         0.0.0.0:*               LISTEN      1022/node-problem-d 
tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN      1221/kubelet        
tcp        0      0 127.0.0.1:10249         0.0.0.0:*               LISTEN      1369/kube-proxy     
tcp        0      0 0.0.0.0:5355            0.0.0.0:*               LISTEN      297/systemd-resolve 
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      330/sshd            
tcp6       0      0 :::30816                :::*                    LISTEN      1369/kube-proxy     
tcp6       0      0 :::4194                 :::*                    LISTEN      1221/kubelet        
tcp6       0      0 :::30472                :::*                    LISTEN      1369/kube-proxy     
tcp6       0      0 :::10250                :::*                    LISTEN      1221/kubelet        
tcp6       0      0 :::5355                 :::*                    LISTEN      297/systemd-resolve 
tcp6       0      0 :::10255                :::*                    LISTEN      1221/kubelet        
tcp6       0      0 :::10256                :::*                    LISTEN      1369/kube-proxy

Two questions:

Given nothing on the node is listening on ports 80 or 443, is the load balancer directing traffic to ports 30472 and 30816?
If the load balancer is accepting traffic on 80/443 and forwarding to 30472/30816, where can I see that configuration? Clicking around the load balancer screens I can't see any mention of ports 30472 and 30816.

user_127255 · Answer

I think I found the answer to my own question - can anyone confirm I'm on the right track?

The network load balancer redirects the traffic to a node in the cluster without modifying the packet - packets for port 80/443 still have port 80/443 when they reach the node.

There's nothing listening on ports 80/443 on the nodes. However kube-proxy has written iptables rules that match packets to the load balancer IP, and rewrite them with the appropriate ClusterIP and port:

You can see the iptables config on the node:

$ iptables-save | grep KUBE-SERVICES | grep loadbalancer                                                                                                            
-A KUBE-SERVICES -d 35.x.y.z/32 -p tcp -m comment --comment "default/contour:http loadbalancer IP" -m tcp --dport 80 -j KUBE-FW-D53V3CDHSZT2BLQV                                                                 
-A KUBE-SERVICES -d 35.x.y.z/32 -p tcp -m comment --comment "default/contour:https loadbalancer IP" -m tcp --dport 443 -j KUBE-FW-J3VGAQUVMYYL5VK6  

$ iptables-save | grep KUBE-SEP-ZAA234GWNBHH7FD4
:KUBE-SEP-ZAA234GWNBHH7FD4 - [0:0]
-A KUBE-SEP-ZAA234GWNBHH7FD4 -s 10.60.0.30/32 -m comment --comment "default/contour:http" -j KUBE-MARK-MASQ
-A KUBE-SEP-ZAA234GWNBHH7FD4 -p tcp -m comment --comment "default/contour:http" -m tcp -j DNAT --to-destination 10.60.0.30:8080

$ iptables-save | grep KUBE-SEP-CXQOVJCC5AE7U6UC
:KUBE-SEP-CXQOVJCC5AE7U6UC - [0:0]
-A KUBE-SEP-CXQOVJCC5AE7U6UC -s 10.60.0.30/32 -m comment --comment "default/contour:https" -j KUBE-MARK-MASQ
-A KUBE-SEP-CXQOVJCC5AE7U6UC -p tcp -m comment --comment "default/contour:https" -m tcp -j DNAT --to-destination 10.60.0.30:8443

An interesting implication is the the nodePort is created but doesn't appear to be used. That matches this comment in the kube docs:

Google Compute Engine does not need to allocate a NodePort to make LoadBalancer work

It also explains why GKE creates an automatic firewall rule that allows traffic from 0.0.0.0/0 towards ports 80/443 on the nodes. The load balancer isn't rewriting the packets, so the firewall needs to allow traffic from anywhere to reach iptables on the node, and it's rewritten there.

user_78722 · Answer

To understand LoadBalancer services, you first have to grok NodePort services. The way those work is that there is a proxy (usually actually implemented in iptables or ipvs now for perf but that's an implementation detail) on every node in your cluster, and when create a NodePort service it picks a port that is unused and sets every one of those proxies to forward packets to your Kubernetes pod. A LoadBalancer service builds on top of that, so on GCP/GKE it creates a GCLB forwarding rule mapping the requested port to a rotation of all those node-level proxies. So the GCLB listens on port 80, which proxies to some random port on a random node, which proxies to the internal port on your pod.

The process is a bit more customizable than that, but that's the basic defaults.

How do GCP load balancers route traffic to GKE services?

2 Answers