Node not enough temp storage for ephemeral storage

user_5379182 0
Viewed 149

I am deploying a Pod to a Standard_NC12s_v3 node on Azure Kubernetes. On the microsoft webpage this node is described to have 1474Gb temp storage (SSD).

In the pod spec, I set the ephemeral-storage to be at least 100Gi (see resource description below). However, when I run $ df -h in the pod, the ephemeral storage (of type emptyDir) has a size of 124G. I would have expected it to have 100G like I requested. The overlay storage I would have expected to be close to 1474Gb (nodes SSD disk size)

enter image description here

My requirement is to have a temporary volume on the pod which is deleted when the pod dies. The reason for it is that I need fast disk IO instead of network IO when storing on a persistent volume.

Pod resource description:

kind: Pod
apiVersion: v1
metadata:
  name: {{ .Values.name }}
  labels:
    for: devs
spec:
  containers:
  - name: {{ .Values.name }}
    image: "{{ .Values.image.acr }}/{{ .Values.image.name }}:{{ .Values.image.tag }}" 
    command: ["/bin/sleep", "3650d"]
    imagePullPolicy: {{ .Values.image.pullPolicy }}
    resources:
      requests:
        cpu: "1"
        memory: 12G
        ephemeral-storage: 100Gi
      limits:
        cpu: "2"
        memory: 24G
        ephemeral-storage: 300Gi
    volumeMounts:
    - mountPath: {{ .Values.pvc.mount }}
      name: volume
    - mountPath: /cache
      name: cache-volume
  restartPolicy: Always
  volumes:
    - name: volume
      persistentVolumeClaim:
        claimName: {{ .Values.pvc.name }}
    - name: cache-volume
      emptyDir: {}
  {{- if .Values.tolerations }}
  tolerations:
  {{- toYaml .Values.tolerations | nindent 4 }}
  {{- end }}
kubernetes azure
1 Answers

Kubernetes supports several different kinds of ephemeral volumes for different purposes emptyDir is one of those.

emptyDir is itself a temporary storage in pod and delete when pod dies.

Commonly used as temporary space for a pod. All containers within a pod can access the data on the volume. Data written to this volume type persists only for the lifespan of the pod. Once you delete the pod, the volume is deleted. This volume typically uses the underlying local node disk storage, though it can also exist only in the node's memory.

The ephemeral storage (of type emptyDir) has a size of 124G. I would have expected it to have 100G like I requested

emptyDir is managed by kubelet on each node. emptyDir: empty at Pod startup, with storage coming locally from the kubelet base directory (usually the root disk) or RAM

You can also refer this github discussion where different user has reported same kind of issue.

Conculsion : ephemeral-storage varies from its require size because you have filesystem running on node is using as empheral storage,The kubelet also writes node-level container logs into the first filesystem, and treats these similarly to ephemeral local storage. That might be reason for exteding the size of empheral storage and as there is also set limit of 300GB for empheral storage.

user_159693340
Related