kube-router assigning ClusterIPs outside of default subnet

user_2073529 0
Viewed 516

I have in my home lab a default installation of Kubernetes with kube-router as the network provider. kube-router is, as default, set as the service proxy. I have not set an explicit service-cluster-ip-network in my kube-controller-manager, so kube-router should be assigning service cluster IPs only within the default 10.96.x.x/16 subnet. However, I am regularly getting service cluster IPs anywhere within the larger 10.x.x.x./8 subnet. I am at a loss where/why it's not remaining within 10.96.x.x. Ideas? Thanks!

kubernetes networking
2 Answers

TL;DR

Your Kubernetes cluster is behaving correctly.

By default (if not specified otherwise) using kubeadm to provision your cluster, the --service-cidr is set to 10.96.0.0/12.

ClusterIP address like 10.110.15.13 would be included in the above mentioned network (10.96.0.0/12).

I've provided more explanation below:

Subnetting

If you use one of the available online IP calculators you will be seeing exact same situation like the one included below:

CIDR 10.96.0.0/12
Subnet mask 255.240.0.0
Network address (first) 10.96.0.0
Broadcast address (last) 10.111.255.255
First useable address 10.96.0.1
Last useable address 10.111.255.254
Number of hosts allocatable 1048574

By above diagram you can see that the Service IP range would be following:

  • 10.96.0.1-10.111.255.254

This would make IP's like: 10.104.5.2, 10.110.15.13 be in range of above network.

Kubernetes --service-cidr

As said earlier if you don't specify the --service-cidr when using $ kubeadm init it will be set to default 10.96.0.0/12.

Following the official documentation of kubeadm:

--service-cidr string     Default: "10.96.0.0/12"
Use alternative range of IP address for service VIPs.

-- Kubernetes.io: Docs: Reference: Setup tools: Kubeadm: Kubeadm init: Options

If you provision the cluster without this parameter, you will be able to see it configured in the:

  • kube-apiserver:
$ kubectl get pods -n kube-system   kube-apiserver-kubernetes-NODE_NAME -o yaml | grep "service-cluster-ip-range" 
    - --service-cluster-ip-range=10.96.0.0/12
  • kube-controller-manager
$ kubectl get pods -n kube-system   kube-controller-manager-kubernetes-NODE_NAME -o yaml | grep "service-cluster-ip-range" 
    - --service-cluster-ip-range=10.96.0.0/12

Kube-router

It's also explicitly stated in the kube-router's source code:

func NewKubeRouterConfig() *KubeRouterConfig {
  return &KubeRouterConfig{
      // SKIPPED 
      ClusterIPCIDR:                  "10.96.0.0/12",
       // SKIPPED
  }
}

-- Github.com: Cloudnativelabds: Kube-router: Pkg: Options: Options.go: Line 73

  fs.StringVar(&s.ClusterIPCIDR, "service-cluster-ip-range", s.ClusterIPCIDR,
      "CIDR value from which service cluster IPs are assigned. Default: 10.96.0.0/12")

-- Github.com: Cloudnativelabds: Kube-router: Pkg: Options: Options.go: Line 187

It's also referenced in the user guide.

user_122571340

For this issue, I was able to find the file /etc/kubernetes/manifests/kube-apiserver.yaml on the master node and correct the service-cluster-ip-range line.

In my case, it appears that a kubeadm command to install the descheduler modified the line.

user_177054280
Related