How to install AWS Load Balancer Controller in a CDK project

Viewed 1282

I am trying to write a high-level CDK construct that can be used to deploy Django applications with EKS. I have most of the k8s manifests defined for the application, but I am struggling with the Ingress part. Looking into different options, I have decided to try installing the AWS Load Balancer Controller (https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.2/deploy/installation/). Their documentation has instructions for installing the controller using the AWS CLI and the eksctl CLI tool, so I'm working on trying to translate these into CDK code. Here's what I have so far:

import * as ec2 from '@aws-cdk/aws-ec2';
import * as eks from '@aws-cdk/aws-eks';
import * as iam from '@aws-cdk/aws-iam';
import * as cdk from '@aws-cdk/core';
import { ApplicationVpc } from './vpc';
var request = require('sync-request');


export interface DjangoEksProps {
  readonly vpc: ec2.IVpc;
}


export class DjangoEks extends cdk.Construct {

  public vpc: ec2.IVpc;
  public cluster: eks.Cluster;

  constructor(scope: cdk.Construct, id: string, props: DjangoEksProps) {
    super(scope, id);

    this.vpc = props.vpc;


    // allow all account users to assume this role in order to admin the cluster
    const mastersRole = new iam.Role(this, 'AdminRole', {
      assumedBy: new iam.AccountRootPrincipal(),
    });

    this.cluster = new eks.Cluster(this, "MyEksCluster", {
      version: eks.KubernetesVersion.V1_19,
      vpc: this.vpc,
      mastersRole,
      defaultCapacity: 2,
    });

    // Adopted from comments in this issue: https://github.com/aws/aws-cdk/issues/8836
    const albServiceAccount = this.cluster.addServiceAccount('aws-alb-ingress-controller-sa', {
      name: 'aws-load-balancer-controller',
      namespace: 'kube-system',
    });

    const awsAlbControllerPolicyUrl = 'https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.2.0/docs/install/iam_policy.json';
    const policyJson = request('GET', awsAlbControllerPolicyUrl).getBody('utf8');
    ((JSON.parse(policyJson))['Statement'] as any[]).forEach(statement => {
      albServiceAccount.addToPrincipalPolicy(iam.PolicyStatement.fromJson(statement))
    })

    // This is where I am stuck
    // https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.2/deploy/installation/#add-controller-to-cluster

    // I tried running this
    // kubectl apply -k "github.com/aws/eks-charts/stable/aws-load-balancer-controller//crds?ref=master"

    this.cluster.addHelmChart('aws-load-balancer-controller-helm-chart', {
      repository: 'https://aws.github.io/eks-charts',
      chart: 'eks/aws-load-balancer-controller',
      release: 'aws-load-balancer-controller',
      version: '2.2.0',
      namespace: 'kube-system',
      values: {
        clusterName: this.cluster.clusterName,
        serviceAccount: {
          create: false,
          name: 'aws-load-balancer-controller',
        },
      },
    });
  }
}

Here are the errors I am seeing in CDK when I do cdk deploy:

Received response status [FAILED] from custom resource. Message returned: Error: b'WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /tmp/kubeconfig\nWARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /tmp/kubeconfig\nRelease "aws-load-balancer-controller" does not exist. Installing it now.\nError: chart "eks/aws-load-balancer-controller" version "2.2.0" not found in https://aws.github.io/eks-charts repository\n' Logs: /aws/lambda/DjangoEks-awscdkawseksKubectlProvi-Handler886CB40B-6yld0A8rw9hp at invokeUserFunction (/var/task/framework.js:95:19) at processTicksAndRejections (internal/process/task_queues.js:93:5) at async onEvent (/var/task/framework.js:19:27) at async Runtime.handler (/var/task/cfn-response.js:48:13) (RequestId: ec066bb2-4cc1-48f6-8a88-c6062c27ed0f)

and a related error:

Received response status [FAILED] from custom resource. Message returned: Error: b'error: no objects passed to create\n' Logs: /aws/lambda/DjangoEks-awscdkawseksKubectlProvi-Handler886CB40B-6yld0A8rw9hp at invokeUserFunction (/var/task/framework.js:95:19) at processTicksAndRejections (internal/process/task_queues.js:93:5) at async onEvent (/var/task/framework.js:19:27) at async Runtime.handler (/var/task/cfn-response.js:48:13) (RequestId: fe2c4c04-4de9-4a71-b18a-ab5bc91d180a)

The CDK EKS docs say that addHelmChart will install the provided Helm Chart with helm upgrade --install.

The AWS Load Balancer Controller installation instructions also say:

Install the TargetGroupBinding CRDs if upgrading the chart via helm upgrade.

kubectl apply -k "github.com/aws/eks-charts/stable/aws-load-balancer-controller//crds?ref=master"

I'm not sure how I can do this part in CDK. The link to github.com/aws/eks-charts/stable/aws-load-balancer-controller//crds?ref=master gives a 404, but that command does work when I run it against my EKS cluster, and I can install those CRDs. Running the deploy command after manually installing those CRDs also fails with the same message.

I think the error in my code comes from the HelmChartOptions that I pass to the addHelmChart command, and I have tried several options, and referenced similar CDK projects that install Helm charts from the same repo, but I keep getting failures.

Is anyone else installing the AWS Load Balancer Controller with CDK like I am trying to do here? One project that I have been trying to reference is https://github.com/neilkuan/cdk8s-aws-load-balancer-controller.

There is also discussion in this GitHub issue: https://github.com/aws/aws-cdk/issues/8836 that might help as well, but a lot of the discussion is around cert manager manager which doesn't seem to be relevant for what I'm trying to do.

2 Answers
Related