Error - "Transport endpoint is not connected" while using s3fs with kubernetes when pod fails

user_2929240 0
Viewed 1851

I have an EKS cluster where having one daemon which mounts a s3 bucket to all pods.

Whenever there is some issue or pod restarts, then the mount volume is not accessible and throws the below error.

Transport endpoint is not connected

For solving this error, I have to manaully unmount the volume and restart the daemon.

umount /mnt/data-s3-fuse

What could be the permanent solution for this issue?

My Daemon file

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  labels:
    app: s3-provider
  name: s3-provider
  namespace: airflow
spec:
  template:
    metadata:
      labels:
        app: s3-provider
    spec:
      containers:
      - name: s3fuse
        image: image
        lifecycle:
          preStop:
            exec:
              command: ["/bin/sh","-c","umount -f /opt/airflow/dags"]
        securityContext:
          privileged: true
          capabilities:
            add:
            - SYS_ADMIN
        # use ALL  entries in the config map as environment variables
        envFrom:
        - configMapRef:
            name: s3-config
        volumeMounts:
        - name: devfuse
          mountPath: /dev/fuse
        - name: mntdatas3fs
          mountPath: /opt/airflow/dags:shared
      volumes:
      - name: devfuse
        hostPath:
          path: /dev/fuse
      - name: mntdatas3fs
        hostPath:
          path: /mnt/data-s3-fuse

and my pod yaml is

apiVersion: v1
kind: Pod
metadata:
  name: test-pd
  namespace: airflow
spec:
  containers:
  - image: nginx
    name: s3-test-container
    securityContext:
      privileged: true
    volumeMounts:
    - name: mntdatas3fs
      mountPath: /opt/airflow/dags:shared
    livenessProbe:
      exec:
        command: ["ls", "/opt/airflow/dags"]
      failureThreshold: 3
      initialDelaySeconds: 10
      periodSeconds: 5
      successThreshold: 1
      timeoutSeconds: 1
  volumes:
  - name: mntdatas3fs
    hostPath:
      path: /mnt/data-s3-fuse

I am using the below code for the s3 kubernetes fuse.

https://github.com/freegroup/kube-s3

amazon-eks s3fs kubernetes amazon-s3
2 Answers

Ok, I think I solved it. It seems like sometimes the pod looses the connection, resulting in "Transport not connected". The workaround I found to fix this is to add an init container, that tries to unmount the folder before. That seems to fix the issue. Note, you want to mount a higher-level folder, so you have access to the node. Will let it run and see if it comes back, it seems to have fixed the issue once here:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    app: s3-provider
  name: s3-provider
spec:
  selector:
    matchLabels:
      app: s3-provider
  template:
    metadata:
      labels:
        app: s3-provider
    spec:
      initContainers:
      - name: init-myservice
        image: bash
        command: ['bash', '-c', 'umount -l /mnt/data-s3-fs/root ; true']
        securityContext:
          privileged: true
          capabilities:
            add:
            - SYS_ADMIN
        # use ALL  entries in the config map as environment variables
        envFrom:
        - configMapRef:
            name: s3-config
        volumeMounts:
        - name: devfuse
          mountPath: /dev/fuse
        - name: mntdatas3fs-init
          mountPath: /mnt:shared
      containers:
      - name: s3fuse
        image: 963341077747.dkr.ecr.us-east-1.amazonaws.com/kube-s3:1.0
        imagePullPolicy: Always
        lifecycle:
          preStop:
            exec:
              command: ["bash", "-c", "umount -f /srv/s3-mount/root"]
        securityContext:
          privileged: true
          capabilities:
            add:
            - SYS_ADMIN
        # use ALL  entries in the config map as environment variables
        envFrom:
        - configMapRef:
            name: s3-config
        env:
        - name: S3_BUCKET
          value: s3-mount
        - name: MNT_POINT
          value: /srv/s3-mount/root
        - name: IAM_ROLE
          value: none
        volumeMounts:
        - name: devfuse
          mountPath: /dev/fuse
        - name: mntdatas3fs
          mountPath: /srv/s3-mount/root:shared
      volumes:
      - name: devfuse
        hostPath:
          path: /dev/fuse
      - name: mntdatas3fs
        hostPath:
          type: DirectoryOrCreate
          path: /mnt/data-s3-fs/root
      - name: mntdatas3fs-init
        hostPath:
          type: DirectoryOrCreate
          path: /mnt
user_3112680

For me, the solution was having a preStop hook event to unmount the path before the pod exits:

    containers:
      - name: aws-sync
        lifecycle:
          preStop:
            exec:
              command: ['bash', '-c', 'umount -l /mounted/path; true']
user_70434830
Related