Is it possible to delete and re-create GKE nodepool with new machine type that is managed through terraform?

user_7410183 0
Viewed 1196

I want to change the machine type on my gke nodepool to better match my cpu vs memory usage, but I am having a lot of trouble with getting terraform to delete this nodepool and re-create it. I am locked into having a default nodepool because of the module that has been used in the past to create the cluster. The cluster is in a shared module with the nodepool, so I cannot permanently delete the nodepool through terraform without also deleting the cluster which would affect everybody that is expecting the cluster to stay available.

So my solution was to create an additional and temporary nodepool, migrate all pods to it, cordon and drain the default nodepool, then through terraform, change the nodepool's machine type so that it would recreate without affecting any running deployments, pods, etc. However, terraform did not attempt to delete the nodepool, only to re-create it. Therefore it failed with a 409, nodepool already exists.

My question is, can I delete a nodepool manually - through gcloud commands or other such methods - and then re-run terraform and hopefully not experience the 409 (nodepool already exists) error? Are there any consequences this could have on the terraform state file? Would terraform fail completely if I deleted a resource (the nodepool) that it was expecting to exist?

Note - I did my best to include all information, but if there's more info needed please let me know and I will attempt to edit this and add more info. Thanks.

google-kubernetes-engine terraform
2 Answers

If your issue is the default terraform behaviour of destroying resources before creating new resources you could try to use the 'lifecycle' meta-argument in your terraform configuration.

resource "google_container_node_pool" "example" {
  # ...

  lifecycle {
    create_before_destroy = true
  }
}

This would ensure that your node pool which you want to replace stays up until the replacement pool with new machine type has been created.

user_150143050

I have found an example on how to Migrate workloads to different machine types Basically you did the correct actions.

Basically is what you already did:

My solution was to create an additional and temporary nodepool, migrate all pods to it, cordon and drain the default nodepool

But if you want to re-apply your TF after doing manual operations, then you are probably going to have a lot of errors since TF saves a status.

what I think you would have to do is all the steps above but with TF so in that way it does not lose consistency with its status.

Especially since the image change is not such a transparent operation for GKE, that's why the documentation recommends creating a new nodepool.

Take in consideration that Terraform will not let you manage your cluster (cordon, drain, etc.) that must be by hand.

user_140048440
Related