Kubernetes nfs provider selfLink was empty

user_4224464 0
Viewed 11742

Yesterday I created a new kubernetes cluster (v1.20.1, on prem) and I wanted to add NFS provisioning. The only NFS provisioner available (and still maintained) seems to be https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner.

It does say to use your own provisioner and the default (quay.io/external_storage/nfs-client-provisioner:latest) is two years old, but I don't have my own provisioner.

When I follow the deployment guide without the helm chart and check the nfs-client-provisioner log I see the following:

I1220 22:20:44.160099       1 leaderelection.go:185] attempting to acquire leader lease  default/fuseim.pri-ifs...
E1220 22:21:01.598029       1 event.go:259] Could not construct reference to: '&v1.Endpoints{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"fuseim.pri-ifs", GenerateName:"", Namespace:"default", SelfLink:"", UID:"c852ca40-471f-4019-a099-d72d32555022", ResourceVersion:"134579", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63744022156, loc:(*time.Location)(0x1956800)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string{"control-plane.alpha.kubernetes.io/leader":"{\"holderIdentity\":\"nfs-client-provisioner-5999484954-n4tj7_94db294f-4261-11eb-9b30-c64536689731\",\"leaseDurationSeconds\":15,\"acquireTime\":\"2020-12-20T01:21:01Z\",\"renewTime\":\"2020-12-20T01:21:01Z\",\"leaderTransitions\":2}"}, OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Subsets:[]v1.EndpointSubset(nil)}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'LeaderElection' 'nfs-client-provisioner-5999484954-n4tj7_94db294f-4261-11eb-9b30-c64536689731 became leader'
I1220 22:21:01.598123       1 leaderelection.go:194] successfully acquired lease default/fuseim.pri-ifs
I1220 22:21:01.598198       1 controller.go:631] Starting provisioner controller fuseim.pri/ifs_nfs-client-provisioner-5999484954-n4tj7_94db294f-4261-11eb-9b30-c64536689731!
I1220 22:21:01.709535       1 controller.go:680] Started provisioner controller fuseim.pri/ifs_nfs-client-provisioner-5999484954-n4tj7_94db294f-4261-11eb-9b30-c64536689731!
I1220 22:21:01.717419       1 controller.go:987] provision "default/test-claim" class "managed-nfs-storage": started
E1220 22:21:01.720318       1 controller.go:1004] provision "default/test-claim" class "managed-nfs-storage": unexpected error getting claim reference: selfLink was empty, can't make reference
I1220 22:36:01.615073       1 controller.go:987] provision "default/test-claim" class "managed-nfs-storage": started
E1220 22:36:01.618195       1 controller.go:1004] provision "default/test-claim" class "managed-nfs-storage": unexpected error getting claim reference: selfLink was empty, can't make reference

Is this a problem with my cluster or with this provisioner? I really have no clue.

I can also just ditch this provisioner, or even NFS, for something else. I just need storage per pod instance to work, e.g. volumeClaimTemplates, stored outside of my kubernetes cluster. If anyone has any suggestions, please let me know!

Thanks in advance, Hendrik

kubernetes nfs storage
4 Answers

You are seeing this error beacause of this: KEP-1164: Deprecate and Remove SelfLink

Quote from mentioned KEP:

In v1.16, we will deprecate the SelfLink field in both ObjectMeta and ListMeta objects by: documenting in field definition that it is deprecated and is going to be removed adding a release-note about field deprecation We will also introduce a feature gate to allow disabling setting SelfLink fields and opaque the logic setting it behind this feature gate.

In v1.20 (12 months and 4 release from v1.16) we will switch off the feature gate which will automatically disable setting SelfLinks. However it will still be possible to revert the behavior by changing value of a feature gate.

In v1.21, we will get rid of the whole code propagating those fields and fields themselves. In the meantime, we will go over places referencing that field (see below) and get rid of those too.

As you see, you can enable it back with featureGate: RemoveSelfLink=false although its not recommented and SelfLink will be permamently removed in v1.21

Also check this issue on github: Using Kubernetes v1.20.0, getting "unexpected error getting claim reference: selfLink was empty, can't make reference

user_122010840

I resolved this by using gcr.io/k8s-staging-sig-storage/nfs-subdir-external-provisioner:v4.0.0 as suggested in the thread from https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner/issues/25 which Matt had linked.

To use with Helm values file:

image:
    repository: gcr.io/k8s-staging-sig-storage/nfs-subdir-external-provisioner
    tag: v4.0.0

To use with Terraform Helm provider:

  set {
    name  = "image.repository"
    value = "gcr.io/k8s-staging-sig-storage/nfs-subdir-external-provisioner"
  }
  set {
    name  = "image.tag"
    value = "v4.0.0"
  }

I used the specific one the post suggested, it is likely though that their most recent image would have additional fixes so might be prudent to use that instead.

user_1749210

I am seeing the same problem here, but its not only about the selflink provider. I am seeing the issue that since installing the latest version (v1.20.1) I have the problem that my 'testclaim' pvc is no longer able to claim storage from the storageclass (managed-nfs-storage), which worked fine before 1.20.1

So, as a sanity check, I tested this with both v1.17.16 and v1.19.6 and in both versions the status of the testclaim immediately switched to 'Bound', but with v1.20.1 the pvc remains in the status 'Pending' and never changes to 'Bound'. Can anyone shed some light on this issue?

Tx!

Bart

user_148596950

Use https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner/

helm repo add nfs-subdir-external-provisioner \
  https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner/
helm repo update
helm install nfs-provisioner \
  nfs-subdir-external-provisioner/nfs-subdir-external-provisioner \
  --set nfs.server=<server-ip> \
  --set nfs.path=/path/to/nfs/storage \
  --set storageClass.name=nfs-provisioner

Add storageClassName=nfs-provisioner in PVC spec

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-pvc
spec:
  storageClassName: nfs-provisioner
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 20Gi

The below approach is deprecated but still if you must use it, add --feature-gates=RemoveSelfLink=false flag in the API server static manifest /etc/kubernetes/manifests/kube-apiserver.yaml like so

spec:
  containers:
  - command:
    - kube-apiserver
    - --feature-gates=RemoveSelfLink=false
user_78052540
Related