Here are the common things to check and do when you can't access services outside of your GKE cluster:
- Check if you can access the service from the GKE VM directly by SSHing into the VM and then doing ping/curl to verify connectivity
- Verify that the ip-masq-agent is running on the GKE cluster and that it's configured correctly
- Configure Cloud NAT for the network when you're using Private Clusters so you can still access internet resources
See more details for each these steps below:
Verifying connectivity from the GKE node itself
SSH into the node and run ping / curl by running:
gcloud compute ssh my-gke-node-name --tunnel-through-iap
curl https://facebook.com
ping my-network-printer
Verify ip-masq-agent configuration
Check if the ip-masq-agent Daemonset is running:
kubectl get ds ip-masq-agent -n kube-system
Verify that the ip-masq-agent configuration is set to ensure all RFC1918 addresses get masqueraded except the GKE node CIDR and pod CIDR:
kubectl describe configmaps/ip-masq-agent -n kube-system
Note the default configuration of ip-masq-agent most of the time has too many RFC1918 addresses included in the nonMasqueradeCIDRs setting. You need to ensure your external network printer isn't included in any of nonMasqueradeCIDRs ranges.
If it is included or the no CIDRs are set, then you should set the nonMasqueradeCIDRs to only include the GKE node CIDR and the GKE pod CIDR for your cluster. You can edit the configmap by running:
kubectl edit configmap ip-masq-agent --namespace=kube-system
More docs on GKE ip-masq-agent here: https://cloud.google.com/kubernetes-engine/docs/how-to/ip-masquerade-agent#edit-ip-masq-agent-configmap