I have a Prometheus server deployment running inside EKS cluster. The EBS volume attached to prometheus deployment is un-encrypted. I want to encrypt the volume attached to the prometheus server deployment. I don't want to suffer data loss or maybe minimum data loss. Challenges foreseen are with process of creating encrypted volume and attaching it to the prometheus deployment since time taken for that process would be too long maybe for 600GB of data. Can anyone provide any suggestion, it would be great if someone could provide some sort of help.
1 Answers
I think doing this in place is going to be extremely challenging.
What you could do is boot a second prometheus pod, backed by an encrypted PVC, and configure the first prometheus to remote-write to the second instance.
If you set up the constraints on your cluster nodes correctly via taints and tolerations, you can ensure both prometheus pods run on the same node. You will then be able to ssh in to the eks node, find the two PVC volumes as local filesystem mounts, and cp -R from the source unencrypted volume to the target encrypted volume.
This should allow you to shift the data with no loss.
While on the subject of prometheus - take a look at VictoriaMetrics - it is a near-100% compatible drop-in for prometheus which uses less memory and is much more io and cpu efficient. These are major wins if you need prometheus in a EKS environment.
Related