When creating a new node pool in Azure Kubernetes Services, the default maximum number of Pods per Node differs between CNI and Kubenet (and the tool used to create the node pool). According to the docs, the default setting is usually 30 with CNI, but 110 with Kubenet.
Why does CNI have a lower default setting and what are the downsides of increasing the actual value, e.g., to 110, like with Kubenet?
110 is the default size defined by Kubernetes upstream. With Kubnet only the nodes get an IP on your Azure subnet. All the pods get IPs from a logical network on the nodes and they are using NAT (iptables) to communicate with the Azure Network.
When you use Azure CNI, Azure pre-allocates IPs in your subnet. You can read this here:
Each node is configured with a primary IP address. By default, 30 additional IP addresses are pre-configured by Azure CNI that are assigned to pods scheduled on the node.
This means for every node with max pods 30 you would need 31 addresses free in your subnet. If your subnet is to small, you could not add any new node bcs Azure needs those 31 IP addresses to add this node.
I think max pods 30 is more like a security value bcs people tend to use /24 subnets. With max pods 110 you could only add 2 nodes to this subnet. If your AKS is running, you cannot change the subnet or the max pod size - this would require a new AKS to be created.
There is no downside to using max pods 110 on your nodes except that you need to size your subnet accordingly and you would need more planning. We are mostly using /16 vnets with /21 subnets with max pods 110 on our AKS clusters:
Clusters configured with Azure CNI networking require additional planning. The size of your virtual network and its subnet must accommodate the number of pods you plan to run and the number of nodes for the cluster.
