Avoiding a race condition in cdk bootstrap - concurrent runs of test and deploy pipeline

user_3969918 0
Viewed 1205

I have a production upgrade test pipeline that is run on every PR that will deploy a copy of production to a test account and then upgrade that with what's on the feature branch. This is done to ensure that the production upgrade is always successful. The steps are:

  1. bootstrap and deploy master branch to a pre-prod AWS account
  • git checkout master
  • cdk bootstrap --force aws://unknown-account/ap-southeast-2
  • cdk deploy --all --require-approval never --change-set-name "ci-${GITHUB_RUN_ID}"
  1. bootstrap and deploy feature branch to same account
  • git checkout feature
  • cdk bootstrap aws://unknown-account/ap-southeast-2
  • cdk deploy --all --require-approval never --change-set-name "ci-${GITHUB_RUN_ID}"

Yesterday we had an issue when bumping aws-cdk to 1.108.0. The second step above failed to deploy, but the step still successfully bootstrapped the AWS account to that version 6. (5 -> 6)

When re-running the job a second time we received issues like below when running the first step in pipeline:

 ❌   Environment aws://unknown-account/ap-southeast-2 failed bootstrapping: Error: Not downgrading existing bootstrap stack from version '6' to version '5'. Use --force to force.

We got around this error by using the --force flag when bootstrapping on the first step in pipeline in order to fully recreate the production environment in our test account.

But now because we are using --force, we are bootstrapping the account every single time we run the pipeline which has inadvertently created a race condition between all concurrent PR's and the bootstrap stack in our test account, receiving errors like below whenever 2 test jobs are run at the same time.

 ❌  Environment aws://unknown-account/ap-southeast-2 failed bootstrapping: InvalidChangeSetStatus: Cannot delete ChangeSet in status CREATE_IN_PROGRESS

What is the best way to either

  1. Specifically allow downgrades in cdk bootstrap without using --force flag
  2. Failing that, avoid this race condition?

We have looked into perhaps using the --qualifier flag on cdk bootstrap to create a bootstrap for every test run but thinking that due to this not being implemented https://github.com/aws/aws-cdk/issues/986 that this might not be so great an idea.

aws-cdk
1 Answers

we've got around this by adding a || true to the script so even if the script returns an error, the step succeeds.

and if a real error does occurr for a new setup in a new account, then the following cdk deploy step fails anyway as the previous step of bootstraping has failed, which does allow us to investigate the issue from logs.

user_26552960
Related