Airflow: Unable to get persistent logs using KubernetesExecutor and PV (official helm chart)

user_11216435 0
Viewed 1118

I feel a bit like an idiot but I cannot seem to get the logging working on persistent volumes when using the KubernetesExecutor and the freshly released official Helm chart.

After creating a simple PV and PVC manually, I changed the following on the bottom of values.yaml file:

logs:
  persistence:
    # Enable persistent volume for storing logs
    enabled: true
    # Volume size for logs
    size: 100Gi
    # If using a custom storageClass, pass name here
    storageClassName:
    ## the name of an existing PVC to use
    existingClaim: airflow-logs

This process is partly described in the official Helm documentation. Still, the airflow-scheduler pod crashes due to permission errors as i cannot write in the mounted logs folder: logs here.

When the persistent logging is turned off, all is working, except for task logging as these are deleted when the worker-pod is deleted.

Any help towards a solution would be greatly appreciated!

airflow-scheduler airflow kubernetes-helm kubernetes
1 Answers

I assumed that using standard persistent volume approach was the easiest (I am still a k8s novice) I did not expect that setting up one using azure-file-storage-class (SC) was this easy. These mounts can be set up using 777 rights from the SC yaml file, not sure if this is the sole cure as I also set the uid/gid in the SC mount options. Anyhow, all seems to be working perfectly.

As a reference for otheres here my azure-file-sc.yaml:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: airflow-logs
provisioner: kubernetes.io/azure-file
mountOptions:
  - dir_mode=0777
  - file_mode=0777
  - uid=50000
  - gid=0
  - mfsymlinks
  - cache=strict
  - actimeo=30
parameters:
  skuName: Standard_LRS

My azure-file-pvc.yaml:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: airflow-logs
  namespace: airflow
  labels:
    app: airflow-logs
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: airflow-logs
  resources:
    requests:
      storage: 20Gi

The values.yaml is unchanged.

With this, the persistent logging works like a charm in Azure Kubernetes Service (AKS).

Hope this helps others! Dennis

user_112164350
Related