Share using NFS a volume mounted with gcsfuse in Kubernetes

Viewed 62

I've been trying to mount a Google Cloud bucket inside a pod in our (onprem) cluster in order to share that mounted volume using NFS to other pods and PersistentVolumes.

Here there are the configurations:

#!/bin/bash

_start_nfs() {
  exportfs -a   
  rpcbind
  rpc.statd
  rpc.nfsd
  rpc.mountd
  GOOGLE_APPLICATION_CREDENTIALS=/accounts/key.json gcsfuse -o allow_other --dir-mode 777 --uid 1500 --gid 1500 ${BUCKET} /exports
}

_nfs_server_mounts() {
  IFS=':' read -r -a MNT_SERVER_ARRAY <<< "$NFS_SERVER_DIRS"
  for server_mnt in "${MNT_SERVER_ARRAY[@]}"; do
    if [[ ! -d $server_mnt ]]; then
        mkdir -p $server_mnt
    fi
    chmod -R 777 $server_mnt
    cat >> /etc/exports <<EOF
${server_mnt} *(rw,sync,no_subtree_check,all_squash,anonuid=1500,anongid=1500,fsid=$(( ( RANDOM % 100 )  + 200 )))
EOF
  done
  cat /etc/exports
}

_sysconfig_nfs() {
  cat > /etc/sysconfig/nfs <<EOF
#
#
# To set lockd kernel module parameters please see
#  /etc/modprobe.d/lockd.conf
#
# Optional arguments passed to rpc.nfsd. See rpc.nfsd(8)
RPCNFSDARGS=""
# Number of nfs server processes to be started.
# The default is 8.
RPCNFSDCOUNT=${RPCNFSDCOUNT}
#
# Set V4 grace period in seconds
#NFSD_V4_GRACE=90
#
# Set V4 lease period in seconds
#NFSD_V4_LEASE=90
#
# Optional arguments passed to rpc.mountd. See rpc.mountd(8)
RPCMOUNTDOPTS=""
# Port rpc.mountd should listen on.
#MOUNTD_PORT=892
#
# Optional arguments passed to rpc.statd. See rpc.statd(8)
STATDARG=""
# Port rpc.statd should listen on.
#STATD_PORT=662
# Outgoing port statd should used. The default is port
# is random
#STATD_OUTGOING_PORT=2020
# Specify callout program
#STATD_HA_CALLOUT="/usr/local/bin/foo"
#
#
# Optional arguments passed to sm-notify. See sm-notify(8)
SMNOTIFYARGS=""
#
# Optional arguments passed to rpc.idmapd. See rpc.idmapd(8)
RPCIDMAPDARGS=""
#
# Optional arguments passed to rpc.gssd. See rpc.gssd(8)
# Note: The rpc-gssd service will not start unless the
#       file /etc/krb5.keytab exists. If an alternate
#       keytab is needed, that separate keytab file
#       location may be defined in the rpc-gssd.service's
#       systemd unit file under the ConditionPathExists
#       parameter
RPCGSSDARGS=""
#
# Enable usage of gssproxy. See gssproxy-mech(8).
GSS_USE_PROXY="yes"
#
# Optional arguments passed to blkmapd. See blkmapd(8)
BLKMAPDARGS=""
EOF
}

### main ###
_sysconfig_nfs
_nfs_server_mounts
_start_nfs

rpcinfo -p
showmount -e

tail -f /dev/null

The Dockerfile:

FROM centos:7
RUN yum -y install /usr/bin/ps nfs-utils nfs4-acl-tools curl portmap fuse nfs-utils && yum clean all
RUN mkdir -p /exports
ENV RPCNFSDCOUNT=8 \
  NFS_SERVER_DIRS='/usr'
RUN chown nfsnobody:nfsnobody /exports
RUN chmod 777 /exports
ADD setup.sh /usr/local/bin/run_nfs.sh
RUN chmod +x /usr/local/bin/run_nfs.sh
RUN useradd -u 1500 orenes
ADD gcsfuse-0.41.6-1.x86_64.rpm /data/gcsfuse-0.41.6-1.x86_64.rpm
RUN yum install -y /data/gcsfuse-0.41.6-1.x86_64.rpm

# Expose volume
VOLUME ["/exports"]

# expose mountd 20048/tcp and nfsd 2049/tcp and rpcbind 111/tcp
EXPOSE 2049/tcp 20048/tcp 111/tcp 111/udp

ENTRYPOINT ["/usr/local/bin/run_nfs.sh"]

Let's assume that there's a PersistentVolume mounting a Service pointing to the NFS server I described before.

So far I got access denied, unable to receive, etc...

Should I move to a SMB/CIFS sharing? Is there any solution on this? Thanks in advance.

1 Answers

It seems that FUSE and NFS don't get well along. I ended up using SMB and it works flawlessly but you must use the Service ClusterIP instead of Kubernetes's DNS.

The comment from @AviD about using a CSI is so helpful if you need to make it quick.

Again thanks to all.

Related