Redis blocks GKE scaledown with reason: no.scale.down.node.pod.has.local.storage

user_18648249 0
Viewed 265

I have a GKE Cluster current scaled upto multiple nodes, the scale up happened during high load due to a DDOS attack on our services , but now the cluster is unable to scale down due to redis-master and redis-slave, this is ending up costing alot of overhead costs that has become an issue for us now.

The autoscale down shows the error: no.scale.down.node.pod.has.local.storage. I have seen in multiple answers setting the option cluster-autoscaler.kubernetes.io/safe-to-evict to true should solve the issue (GCP Suggests the same) but before i do that i want to know if doing so and scaling down redis-slave instances can cause any data loss? Any suggestion for this would be ideal as currently we are paying over 2x of what is needed.

I also checked the config as well , and i saw there are volumes the redis-master and redis-slaves under volumes and the config yaml also has that as :

volumeClaimTemplates:
  - apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      creationTimestamp: null
      labels:
        app: redis
        component: master
        heritage: Tiller
        release: prod-redis
      name: redis-data
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 20Gi
      volumeMode: Filesystem
kubernetes-pod google-kubernetes-engine kubernetes redis google-cloud-platform
1 Answers

Redis master or slave mostly save the data in to memory for backup and restore purpose it takes snapshot every min or second based on config.

You can checkout the deployment config or configmap if any snapshot option running or not. In Redis terms known as the AOF and RDB backup.

Whenver any Redis POD crash in K8s it starts and restores the data from the above file if present.

So make sure you exec into the POD and check files are present or not.

image: redislabs/redis
args: ["--requirepass", "admin", "--appendonly", "yes", "--save", "900", "1", "--save", "30", "2"]

Check the config or YAML of deployment might be having some options like above.

Ref Document : https://redis.io/topics/persistence

If files are not there you can manually take the backup so if POD crashes it start with old data.

Command to generate the backup in background: BG SAVE

if you have large data in POD things might go wrong, BG SAVE will kick the process to save the data to the filesystem in PVC which can lead to Higher resource requirements and POD will get killed if resource set.

Once data is saved using the background command you can start removing the slaves with their respective PVCs.

So if AOF and RDB are disabled data loss will be there.

Just RDB is also not a good option as it takes periodic backup while AOF is an instant option.

If just RDB there, could be chances it has taken snapshot by night and you remove POD in the morning so lastest data after snapshot is in memory you might not get in snapshot.

user_55258240
Related