Update the node size of a digital ocean kubernetes cluster without replacing the whole cluster

Viewed 346

I successfully maintain a kubernetes cluster in digital ocean throught terraform. The core cluster configuration is the following:

resource "digitalocean_kubernetes_cluster" "cluster" {
  name     = var.name
  region   = var.region
  version  = var.k8s_version
  vpc_uuid = digitalocean_vpc.network.id

  node_pool {
    name       = "workers"
    size       = var.k8s_worker_size
    node_count = var.k8s_worker_count
  }
}

The problem is, I now need to increase the node size (stored in the variable k8s_worker_size).

If I simply change the variable to a new string, the terraform plan results in a full replace of the kubernetes cluster:

digitalocean_kubernetes_cluster.cluster must be replaced

This is not doable in our production environment.

The correct procedure to perform this operation inside digital ocean is to:

  • Create a new node pool, with the required size
  • Use kubectl drain to remove our pods from the 'old' nodes
  • Remove the previous node pool.

Of course, by doing this manually inside the digital ocean console, the terraform state is completely out-of-sync and is therefore unusable.

Is there a way to perform that operation through terraform?

As an alternative options, is it possible to "manually" update the terraform state in order to sync it with the real cluster state after I perform the migration manually?

1 Answers

Is there a way to perform that operation through terraform?

There might be some edge cases where there is a solution to this. Since I am not familiar with kubernetes inside DigitalOcean I can't share a specific solution.

As an alternative options, is it possible to "manually" update the terraform state in order to sync it with the real cluster state after I perform the migration manually?

Yes! Do as you proposed manually and then remove the out-of-sync cluster with

terraform state rm digitalocean_kubernetes_cluster.cluster

from the state. Please visit the corresponding documentation for state rm and update the address if your cluster is in a module etc. Then use

terraform import digitalocean_kubernetes_cluster.cluster <id of your cluster>

to reimport the cluster. Please consult the documentation for importing the cluster for the details. The documentations mentions something around tagging the default node pool.

Related