Unable to create EKS cluster when deploying to private subnets

user_11271048 0
Viewed 1234

My objective is to be able to deploy an EKS cluster to aws (I am using Terraform) while keeping it not accessible from the internet (I want it to be secure).

What I have tried (snippet) and worked:

module "eks_cluster" {
  source                                         = "terraform-aws-modules/eks/aws"
  version                                        = "13.2.1"
  cluster_name                                   = "${var.project_name}-foo-${var.environment}"
  cluster_version                                = "1.18"
  vpc_id                                         = module.vpc.vpc_id
  cluster_enabled_log_types                      = ["api", "audit", "authenticator", "controllerManager", "scheduler"]
  enable_irsa                                    = true
  subnets                                        = [module.vpc.subnet_a_private_id, module.vpc.subnet_b_private_id, module.vpc.subnet_a_public_id]
}

What I have tried (snippet) but did not work:

module "eks_cluster" {
  source                                         = "terraform-aws-modules/eks/aws"
  version                                        = "13.2.1"
  cluster_name                                   = "${var.project_name}-foo-${var.environment}"
  cluster_version                                = "1.18"
  vpc_id                                         = module.vpc.vpc_id
  cluster_enabled_log_types                      = ["api", "audit", "authenticator", "controllerManager", "scheduler"]
  enable_irsa                                    = false
  cluster_create_endpoint_private_access_sg_rule = false
  cluster_endpoint_private_access                = true
  cluster_endpoint_private_access_cidrs          = ["0.0.0.0/0"]
  cluster_endpoint_public_access                 = false
  subnets                                        = [module.vpc.subnet_a_private_id, module.vpc.subnet_b_private_id]
}

However, after the cluster is created, the process times out at the step: module.eks_cluster.null_resource.wait_for_cluster[0] with the following error:

Error: Error running command 'for i in `seq 1 60`; do if `command -v wget > /dev/null`; then wget --no-check-certificate -O - -q $ENDPOINT/healthz >/dev/null && exit 0 || true; else curl -k -s $ENDPOINT/healthz >/dev/null && exit 0 || true;fi; sleep 5; done; echo TIMEOUT && exit 1': exit status 1. Output: TIMEOUT

I am happy to provide further details about other configurations if needed.

terraform amazon-web-services
1 Answers

In your second snippet you've enabled private access and disabled public access:

  cluster_endpoint_private_access                = true
  cluster_endpoint_public_access                 = false

If you look at the Modifying cluster endpoint access table in the AWS docs, this combination results in:

All traffic to your cluster API server must come from within your cluster's VPC or a connected network.

There is no public access to your API server from the internet. Any kubectl commands must come from within the VPC or a connected network. For connectivity options, see Accessing a private only API server.

If you check the source code of terraform-aws-modules/eks/aws here, you can find that module.eks_cluster.null_resource.wait_for_cluster[0] from your error is local-exec which tries to access your cluster.

This obviously fails because there is no internet access to your cluster. You either have to run your terraform on some bastion ec2 instance in same VPC or use VPN between your home/work network network and the VPC.

user_2488230
Related