Upgrade (Deployment + PVC) to Statefulset using helm

So, here is what I did. First, let me brief about my test setup:

1. Have a set of deployments running which are having their own PVC+PV for persistence.
2. New release has all of those deployments converted into statefulsets, thus have different PVC.

Problems:

1. You cannot use the PV from earlier deployment because a PV is bound to a single PVC. You can refer to the PV using only that PVC.

2. So another question is, why can't you mount the old PV using it's PVC to new pod and copy that ? Well, we cannot do that because, by the time the new pods are started, the old pods are released thereby releasing its associated PVC and PV. In my case I could see the PVC going to 'Terminate' state. It probably could be solved with some kind of Reclaim policy, but I am not sure of that.

Solution:

1. Create the PV and its PVC's manually and apply them. The PVC name should match the name that the statefulset would create which is quite straight forward. The PVC name matching for statefulset is very important, it expects it to be in certain format which you can find online.

2. In your new Helm chart, create a pre-upgrade kubernetes Job. This is pre-upgrade hook of Helm which is run just before the actual upgrade of Helm release. So, here you create a container and mount both old and new PV (use the PVC you created manually).

3. Now, this container must run something right to copy the data ? Yeah, for that I created a new configmap and put a script in it which would just copy the data from the old PV to new PV. This configmap is mounted inside the container and the container is made to execute that command.

4. Run the helm upgrade command and see it in action.

A rough example of the K8s pre-upgrade job:

apiVersion: batch/v1
kind: Job
metadata:
  name: {{ .Release.Name }}-pre-upgrade-job
  namespace: {{ .Values.namespace.name }}
  annotations:
    "helm.sh/hook": pre-upgrade
spec:
  template:
    spec:
      containers:

      - name: upgrade82-copy
        image: <your-image>
        command: ["/bin/bash"]
        args: ["-c", "/scripts/upgrade82.sh"]

        volumeMounts:
        - name: old-data
          mountPath: /usr/old
          readOnly: false

        - name: new-data
          mountPath: /usr/new
          readOnly: false

        - name: scripts
          mountPath: /scripts
          readOnly: false

      restartPolicy: Never
      volumes:
        - name: old-data
          persistentVolumeClaim:
            claimName: old-claim << Needs to be hardcoded to what is running

        - name: new-data
          persistentVolumeClaim:
            claimName: new-data-namespace-app-0 << There is a specific format for statefulset PVC

        - name: scripts
          configMap:
           name: upgrade
           defaultMode: 0755

  backoffLimit: 4
  #activeDeadlineSeconds: 200
  ttlSecondsAfterFinished : 100

And "upgrade" above is another Configmap with your copy script.

For some reason, we create the pv/pvc at pre-phase for such Statefulset pods. During the upgrade, we delete the old pv/pvc and using the new, delete the old deployments and create headless service also keep some old services for load balancing. Because pv/pvc only bond once, have to create the new pv.