Unable to access Kubernetes service from one cluster to another (over VPC peerng)

user_15383310 0
Viewed 594

I'm wondering if anyone can help with my issue, here's the setup:

  • We have 2 separate kubernetes clusters in GKE, running on v1.17, and they each sit in a separate project
  • We have set up VPC peering between the two projects
  • On cluster 1, we have 'service1' which is exposed by an internal HTTPS load balancer, we don't want this to be public
  • On cluster 2, we intend on being able to access 'service1' via the internal load balancer, and it should do this over the VPC peering connection between the two projects

Here's the issue: When I'm connected via SSH on a GKE node on cluster 2, I can successfully run a curl request to access https://service1.domain.com running on cluster 1, and get the expected response, so traffic is definitely routing from cluster 2 > cluster 1. However, when I'm running the same curl command from a POD, running on a GKE node, the same curl request times out.

I have run as much troubleshooting as I can including telnet, traceroute etc and I'm really stuck why this might be. If anyone can shed light on the difference here that would be great.

I did wonder whether pod networking is somehow forwarding traffic over the clusters public IP rather than over the VPC peering connection.

google-kubernetes-engine kubernetes cloud cluster-computing networking
4 Answers

As mentioned in one of the answers IP aliases (VPC-native) should work out of the box. If using a route based GKE cluster rather than VPC-native you would need to use custom routes.

As per this document

By default, VPC Network Peering with GKE is supported when used with IP aliases. If you don't use IP aliases, you can export custom routes so that GKE containers are reachable from peered networks.

This is also explained in this document

If you have GKE clusters without VPC native addressing, you might have multiple static routes to direct traffic to VM instances that are hosting your containers. You can export these static routes so that the containers are reachable from peered networks.

user_86724660

So it seems you're not using a "VPC-native" cluster and what you need is "IP masquerading".

From this document: "A GKE cluster uses IP masquerading so that destinations outside of the cluster only receive packets from node IP addresses instead of Pod IP addresses. This is useful in environments that expect to only receive packets from node IP addresses."

You can use ip-masq-agent or k8s-custom-iptables. After this, it will work since it will be like you're making a call from node, not inside of pod.

user_57272680

The problem your facing seems similar to the one mentioned in this SO question, perhaps your pods are using IPs outside of the VPC range and for that reason cannot access the peered VPC?

user_38337730

UPDATE: In Google cloud, I tried to access the service from another cluster which had VPC native networking enabled, which I believe allows pods to use the VPC routing and possibly the internal IPs.

Problem solved :-)

user_153833100
Related