kubernetes change PVC from ReadWriteOnce to ReadWriteMany

Viewed 1524

Is it possible to change an existing PVC to ReadWriteMany from ReadWriteOnce without recreating it? I have an existing database pod and the file system has become read only for some reason. I wanted to connect another pod to the PVC to look at it.

2 Answers

Whilst it is not possible to change persistent volume claim from ReadWriteOnce to ReadWriteMany without destroying the persistent volume claim (they are immutable), there is a procedure.

I am writing this here, since same as others I landed here and the question is written without mentioning avoidance of recreation of the PVC.

Here is the docs in kubernetes itself: https://kubernetes.io/blog/2021/09/13/read-write-once-pod-access-mode-alpha/

The example in the docs is to change access mode to ReadWriteOncePod but the principle is identical.

Below is an example and details that I stumbled upon when doing this myself converting access mode from ReadWriteOnce to ReadWriteMany.

I am converting a PersistentVolume and PersistentVolumeClaim for my grafana helm release which is in grafana namespace.

PeristentVolume name: grafana-pv

PersistentVolumeClaim name: grafana-pvc

Namespace: grafana

Adjust commands accordingly to your use case please.

Backup of existing resources

Just in case anything super bad happens, do at least a backup of existing persistent volume claim. We will need to delete it and that's best practice.

Even if you have the code that generated the original PVC, do a backup.

In case you fail to find actual manifest that generated it, you can recreate it then easily from the backup (once you change the access mode).

kubectl get pvc grafana-pvc -n grafana -o yaml > grafana_pvc_backup.yaml

Change reclaim policy of the persistent volume to Retain

This is required!!! Before you delete the persistent volume claim to avoid a surprise that your data got wiped. The default reclaim policy is Delete and we do not want that to happen.

kubectl patch pv grafana-pv -n grafana -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

Scale down any workloads (deployments, stateful sets etc etc) using the volume

In this step you are removing binds to the persistent volume. Check what is using it and scale it down to 0. No need to delete deployments, just scale them down to no pods running.

kubectl scale --replicas=0 -n grafana deployment grafana

When all workloads using the persistent volume have been scaled down to zero you should see the status of it change from Bound to Released. We are not done yet though.

Delete existing persistent volume claim to be able to free persistent volume

NOTE: Did you back up its manifest just in case?

kubectl delete pvc grafana-pvc -n grafana

Free persistent volume status to become Available

Here we just remove the reference to the deleted persistent volume claim from the persistent volume so that a new persistent volume claim can be set.

kubectl patch pv grafana-pv -n grafana -p '{"spec":{"claimRef":{"uid":""}}}'

You should see that status of the persistent volume changed to Available

Change the persistent volume access mode to ReadWriteMany

Just run this command:

kubectl patch pv grafana-pv -n grafana -p '{"spec":{"accessModes":["ReadWriteMany"]}}'

Recreate persistent volume claim with access mode ReadWriteMany

Here you either use your existing code to create this kubernetes resource with access mode ReadWriteMany or just use the backup we did in first step, edit its spec.accessModes and apply that.

I am just showing what the part of spec that is interesting to us should be, if you are editing existing manifest or backup, you will easily find the place to change it.

spec:
  accessModes:
    - ReadWriteMany

Apply the new persistent volume claim manifest (assuming we saved it to a file called persistent_volume_claim_read_write_many.yaml:

kubectl -n grafana apply -f persistent_volume_claim_read_write_many.yaml

Scale the workloads back up to start using the volume with new access mode.

Adjust number of replicas to what your workloads manifests actually specify.

kubectl scale --replicas=1 -n grafana deployment grafana

The status of you persistent volume should become Bound now.

Important checks

Now all the workloads should start using your persistent volume in ReadWriteMany access mode.

Check all containers and make sure they actually were able to attach the volume correctly.

Restore reclaim policy of your persistent volume

If you want it to go back to what it was (assumed Delete) and its status is Bound (meaning something is using it, and it will not get wiped if reclaim policy is changed to Delete).

kubectl patch pv grafana-pv -n grafana -p '{"spec":{"persistentVolumeReclaimPolicy":"Delete"}}'

That's it. All done. I have carried this out and it works. If there are any mistakes - let me know, I will amend the answer.

Related