How to have a Kubernetes Pod connect to a destination only reachable by a subset of Nodes?

user_793961 0
Viewed 292

I have a Kubernetes v1.13 cluster with Calico + flannel as CNI. All Nodes have a publicly routable ip address and are running Ubuntu 16.04.

Some Nodes are located in a company network, being both located in the LAN and DMZ, and therefore having access to internal services while still being publicly accessible. Others are hosted VMs at a cloud provider.

Cluster

Consider the simplified example above. I want a Kubernetes Pod to access Internal Server C (which is just a regular server and not part of the cluster). I could enforce the Pod to be scheduled on the internal Node B only, but as there is only a low latency and bandwidth required for the connection, and there is way more resources on Node A, I would prefer to use Node B just as some kind of gateway. (Consider several Node Bs, so there is actually no SPOF).

My current approach is to use a DaemonSet with a Node Selector targeting all internal (B) Nodes, defining an HAProxy Pod. Those HAproxy instances can be reached as a Kubernetes Service and forward requests to the internal destination services.

Do you see a better or more straightforward way to realize the connection from a Pod located at any Node to a target that can only be reached by a subset of Nodes?

kubernetes networking
2 Answers

Based on what you say here:

I could enforce the Pod to be scheduled on the internal Node B only, but as there is only a low latency and bandwidth required for the connection, and there is way more resources on Node A, I would prefer to use Node B just as some kind of gateway.

I think what you're looking for is an Ambassador pattern. Basically you would create this kind of containers located in your B zone, and your traffic would go to this containers/pods using a ClusterIP service since it's within the cluster.

Then, these containers would have running a proxy inside them (kind of what you have now on your daemons), that would route the traffic transparently to the regular server you're targeting.

Other links that may be useful could be this from MS or this slideshows (p.42).

If this presents a big advantage against what you have already running I'm not sure, but I do prefer to work with just pods and minimize other components if it's possible.

user_17163100

If you like a network layer solution, trying this:

  1. Pick one B node, find its IP address which is under the same subnet with A. Let's say this IP address b_ip
  2. Add a route to server C(maybe a network) using b_ip as its gateway.

This solution has some benefit, that you don't need to modify HAproxy's config when you need to visit a new service in server C. There's something bad either, the B node you picked in step 1 would be a single point in your system. I think you can use a floating IP as a workaround.

user_13957220
Related