Can I use Azure Container Instances without the VMSS?
In an AKS cluster currently you cannot have virtual nodes without a node pool of type VirtualMachineScaleSets or AvailabilitySet. An AKS cluster has at least one node, an Azure virtual machine (VM) that runs the Kubernetes node components and container runtime. [Reference] Every AKS cluster must contain at least one system node pool with at least one node. System node pools serve the primary purpose of hosting critical system pods such as CoreDNS, kube-proxy and metrics-server. However, application pods can be scheduled on system node pools if you wish to only have one pool in your AKS cluster.
For more information on System Node Pools please check this document.
In fact, if you run kubectl get pods -n kube-system -o wide you will see all the system pods running on the VMSS-backed node pool node including the aci-connector-linux-xxxxxxxx-xxxxx pod which connects the cluster to the virtual node, as shown below:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
aci-connector-linux-859d9ff5-24tgq 1/1 Running 0 49m 10.240.0.30 aks-nodepool1-29819654-vmss000000 <none> <none>
azure-cni-networkmonitor-7zcvf 1/1 Running 0 58m 10.240.0.4 aks-nodepool1-29819654-vmss000000 <none> <none>
azure-ip-masq-agent-tdhnx 1/1 Running 0 58m 10.240.0.4 aks-nodepool1-29819654-vmss000000 <none> <none>
coredns-autoscaler-6699988865-k7cs5 1/1 Running 0 58m 10.240.0.31 aks-nodepool1-29819654-vmss000000 <none> <none>
coredns-d4866bcb7-4r9tj 1/1 Running 0 49m 10.240.0.12 aks-nodepool1-29819654-vmss000000 <none> <none>
coredns-d4866bcb7-5vkhc 1/1 Running 0 58m 10.240.0.28 aks-nodepool1-29819654-vmss000000 <none> <none>
coredns-d4866bcb7-b7bzg 1/1 Running 0 49m 10.240.0.11 aks-nodepool1-29819654-vmss000000 <none> <none>
coredns-d4866bcb7-fltbf 1/1 Running 0 49m 10.240.0.29 aks-nodepool1-29819654-vmss000000 <none> <none>
coredns-d4866bcb7-n94tg 1/1 Running 0 57m 10.240.0.34 aks-nodepool1-29819654-vmss000000 <none> <none>
konnectivity-agent-7564955db-f4fm6 1/1 Running 0 58m 10.240.0.4 aks-nodepool1-29819654-vmss000000 <none> <none>
kube-proxy-lntqs 1/1 Running 0 58m 10.240.0.4 aks-nodepool1-29819654-vmss000000 <none> <none>
metrics-server-97958786-bmmv9 1/1 Running 1 58m 10.240.0.24 aks-nodepool1-29819654-vmss000000 <none> <none>
However, you can deploy Azure Container Instances [How-to] without an AKS cluster altogether. For scenarios where you need full container orchestration, including service discovery across multiple containers, automatic scaling, and coordinated application upgrades, we recommend Azure Kubernetes Service (AKS).
If not can I somehow make this VM visible to my cluster so I can at least use it to provision containers onto and get some value out of it?
Absolutely, you can. In fact if you do a kubectl get nodes and the node from the VMSS-backed node pool (in your case aks-control-xxx-vmss-x) shows STATUS as Ready, then it is available to the kube-scheduler for scheduling workloads. Please check this document.
If you do a kubectl describe node virtual-node-aci-linux you should find the following in the output:
...
Labels: alpha.service-controller.kubernetes.io/exclude-balancer=true
beta.kubernetes.io/os=linux
kubernetes.azure.com/managed=false
kubernetes.azure.com/role=agent
kubernetes.io/hostname=virtual-node-aci-linux
kubernetes.io/role=agent
node-role.kubernetes.io/agent=
node.kubernetes.io/exclude-from-external-load-balancers=true
type=virtual-kubelet
...
Taints: virtual-kubelet.io/provider=azure:NoSchedule
...
In the document that you are following, in the Deploy a sample app section to schedule the container on the virtual node, a nodeSelector and toleration are defined in the Deploy a sample app section as follows:
...
nodeSelector:
kubernetes.io/role: agent
beta.kubernetes.io/os: linux
type: virtual-kubelet
tolerations:
- key: virtual-kubelet.io/provider
operator: Exists
- key: azure.com/aci
effect: NoSchedule
If you remove this part from the Deployment manifest, or do not specify this part in the manifest of a workload that you are deploying, then the corresponding resource(s) will be scheduled on a VMSS-backed node.
Have I just done something wrong?
Maybe you can evaluate the answer to this based on my responses to your earlier questions. However, here's a little more to help you understand:
If a node doesn't have sufficient compute resources to run a requested pod, that pod can't progress through the scheduling process. The pod can't start unless additional compute resources are available within the node pool.
When the cluster autoscaler notices pods that can't be scheduled because of node pool resource constraints, the number of nodes within the node pool is increased to provide the additional compute resources. When those additional nodes are successfully deployed and available for use within the node pool, the pods are then scheduled to run on them.
If your application needs to scale rapidly, some pods may remain in a state waiting to be scheduled until the additional nodes deployed by the cluster autoscaler can accept the scheduled pods. For applications that have high burst demands, you can scale with virtual nodes and Azure Container Instances.
This however does not mean that we can dispense with the VMSS or Availability Set backed node pools.
