How to clean up disk space for Azure Kubernetes Node?

user_1300049 0
Viewed 4223

I followed this tutorial to create a Kubernetes cluster on Azure to run build agents: http://www.chrisjohnson.io/2018/07/07/using-azure-kubernetes-service-aks-for-your-vsts-build-agents/

To recap what is there: a helm chart to do a deployment with a secret and a config map. For this deployment, I created a kubernetes cluster on Azure with all default settings and it is pulling an image from the docker hub with vsts build agent installed.

All was working fine, but recently pods started to be evicted pretty regularly, the message on them is:

Message: Pod The node was low on resource: [DiskPressure].

How can I fix this issue?

kubernetes azure
3 Answers

Either/Or:

  • You upgrade the size of your main node disks with something like this.

  • Check what pods are taking up space. Is it logs? Is it cached data? is it swap? Every application is different so you will have to go case by case.

  • Set local ephemeral storage at the pod level for your workloads so that they don't go over. Pods using a lot will get evicted.

  • Use Persistent Volumes for your workloads, especially some that are not local and just reserved for your applications.

user_29892610

Just an FYI if you don't want to increase your disk space...

You can go to your agent's settings (Organization Settings > Agent Pools > {select you agent} > Settings) and make sure the option 'Delete unused working directory after x days' is checked.

If it's checked you may want to decrease the number of days to clean up unused working directories quicker.

user_112986060

For latest kubernetes version AKS, its containerd there is docker, aks have garbage collector which will clean logs when it reaches 85%. we can clean log manually by going into the node (ssh into the node kubectl debug node/aks-node1-xxxxx-vmss00000x -it --image=mcr.microsoft.com/aks/fundamental/base-ubuntu:v0.0.11)

chroot /host

crictl rmi -prune

Above command will clear the logs , cache, unused images etc...

user_41685040
Related