How do you properly setup Terraform to manage an AWS EKS cluster using cross-account IAM assume role authentication?

Question

We're in the process of setting up some green field AWS infrastructure.

We have at the organisation account level an IAM user that Terraform authenticates as using access keys. We've then setup our Terraform code to "assume role" into our sub accounts within their respective git repos (we have 1 git repo per account). Something like,

provider "aws" {
  assume_role {
    role_arn = "arn:aws:iam::XXXXXXXXXX:role/TerraformCloudRole"
  }
}

We're running into issues setting up an EKS cluster using the terraform-aws-modules/eks/aws module. The cluster creates fine, but we've set manage_aws_auth_configmap = true so we can provide IAM roles/users and manage what they can authenticate against. We're actually seeing multiple errors depending on where we do creates or updates, and some subtle changes to the code. Essentially they are,

Error: The configmap "aws-auth" does not exist
with module.eks_main.module.eks.kubernetes_config_map_v1_data.aws_auth[0]
on .terraform/modules/eks_main.eks/main.tf line 470, in resource "kubernetes_config_map_v1_data" "aws_auth":

Or

Error: Get "http://localhost/api/v1/namespaces/kube-system/configmaps/aws-auth": dial tcp [::1]:80: connect: connection refused
with module.eks_main.module.eks.kubernetes_config_map_v1_data.aws_auth[0]
on .terraform/modules/eks_main.eks/main.tf line 470, in resource "kubernetes_config_map_v1_data" "aws_auth":

We did some Googling and we found this issue. We added a provider and this seemed to solve some issues, specifically using this approach. The reason was because the exec route didn't work for us. It appeared to be trying to execute the AWS CLI command using the base access keys and not the assumed role. But the errors are back when we're making updates to the cluster or trying to run a destroy! It doesn't appear to be picking up the provider for some reason. The latter error above is during the plan phase, not apply.

So to my question. How do we setup Terraform to connect to/manage an EKS cluster properly when AWS assume-role/cross-account is involed?

user_5586782 · Answer

maybe help :

provider "aws" {
  alias  = "main"
  region = var.aws_region
}

data "aws_eks_cluster" "selected" {
  provider = aws.main
  name     = local.eks_cluster_name
}

data "aws_eks_cluster_auth" "selected" {
  provider = aws.main
  name     = local.eks_cluster_name
}

provider "kubernetes" {
  host                   = element(concat(data.aws_eks_cluster.selected[*].endpoint, tolist([""])), 0)
  cluster_ca_certificate = base64decode(element(concat(data.aws_eks_cluster.selected[*].certificate_authority.0.data, tolist([""])), 0))
  exec {
    api_version = "client.authentication.k8s.io/v1alpha1"
    args        = ["token", "-i", element(concat(data.aws_eks_cluster.selected[*].id, tolist([""])), 0)]
    command     = "aws-iam-authenticator"
  }
}

provider "helm" {
  kubernetes {
    host                   = element(concat(data.aws_eks_cluster.selected[*].endpoint, tolist([""])), 0)
    cluster_ca_certificate = base64decode(element(concat(data.aws_eks_cluster.selected[*].certificate_authority.0.data, tolist([""])), 0))
    exec {
      api_version = "client.authentication.k8s.io/v1alpha1"
      args        = ["token", "-i", element(concat(data.aws_eks_cluster.selected[*].id, tolist([""])), 0)]
      command     = "aws-iam-authenticator"
    }
  }
}

I'm not sure this will up to date, but I believe this should help you a lot, you can change the command to tho, as it use aws-iam-authenticator binary

How do you properly setup Terraform to manage an AWS EKS cluster using cross-account IAM assume role authentication?

1 Answers