In CDK, can I wait until a Helm-installed operator is running before applying a manifest?

Viewed 46

I'm installing the External Secrets Operator alongside Apache Pinot into an EKS cluster using CDK. I'm running into an issue that I think is being caused by CDK attempting to create a resource defined by the ESO before the ESO has actually gotten up and running. Here's the relevant code:

// install Pinot
const pinot = cluster.addHelmChart('Pinot', {
    chartAsset: new Asset(this, 'PinotChartAsset', { path: path.join(__dirname, '../pinot') }),
    release: 'pinot',
    namespace: 'pinot',
    createNamespace: true
});


// install the External Secrets Operator
const externalSecretsOperator = cluster.addHelmChart('ExternalSecretsOperator', {
    chart: 'external-secrets',
    release: 'external-secrets',
    repository: 'https://charts.external-secrets.io',
    namespace: 'external-secrets',
    createNamespace: true,
    values: {
        installCRDs: true,
        webhook: {
            port: 9443
        }
    }
});


// create a Fargate Profile
const fargateProfile = cluster.addFargateProfile('FargateProfile', {
    fargateProfileName: 'externalsecrets',
    selectors: [{ 'namespace': 'external-secrets' }]
});


// create the Service Account used by the Secret Store
const serviceAccount = cluster.addServiceAccount('ServiceAccount', {
    name: 'eso-service-account',
    namespace: 'external-secrets'
});
serviceAccount.addToPrincipalPolicy(new iam.PolicyStatement({
    effect: iam.Effect.ALLOW,
    actions: [
        'secretsmanager:GetSecretValue',
        'secretsmanager:DescribeSecret'
    ],
    resources: [
        'arn:aws:secretsmanager:us-east-1:<MY-ACCOUNT-ID>:secret:*'
    ]
}))
serviceAccount.node.addDependency(externalSecretsOperator);


// create the Secret Store, an ESO Resource
const secretStoreManifest = getSecretStoreManifest(serviceAccount);
const secretStore = cluster.addManifest('SecretStore', secretStoreManifest);
secretStore.node.addDependency(serviceAccount);
secretStore.node.addDependency(fargateProfile);


// create the External Secret, another ESO resource
const externalSecretManifest = getExternalSecretManifest(secretStoreManifest)
const externalSecret = cluster.addManifest('ExternalSecret', externalSecretManifest)
externalSecret.node.addDependency(secretStore);
externalSecret.node.addDependency(pinot);

Even though I've set the ESO as a dependency to the Secret Store, when I try to deploy this I get the following error:

Received response status [FAILED] from custom resource. Message returned: Error: b'Error from server (InternalError): error when creating "/tmp/manifest.yaml": Internal error occurred:
failed calling webhook "validate.clustersecretstore.external-secrets.io": Post "https://external-secrets-webhook.external-secrets.svc:443/validate-external-secrets-io-v1beta1-clusterse
cretstore?timeout=5s": no endpoints available for service "external-secrets-webhook"\n'

If I understand correctly, this is the error you'd get if you try to add a Secret Store before the ESO is fully installed. I'm guessing that CDK does not wait until the ESO's pods are running before attempting to apply the manifest. Furthermore, if I comment out the lines the create the Secret Store and External Secret, do a cdk deploy, uncomment those lines and then deploy again, everything works fine.

Is there any way around this? Some way I can retry applying the manifest, or to wait a period of time before attempting the apply?

1 Answers

The addHelmChart method has a property wait that is set to false by default - setting it to true lets CDK know to not mark the installation as complete until of its its K8s resources are in a ready state.

Related