AWS CDK cross-stack references and deployment order

AWS added an official workaround. It allows you to manually create an export in the dependent stack. When you need to remove an export, create it manually, remove its usage, deploy, remove it, and deploy again.

this.exportAttribute(this.bucket.bucketName)

One workaround I found is forcing CDK to split the deployment to two steps. First I deploy the stack that uses the export so it doesn't use it anymore. Then I deploy the stack that creates the export to remove the export after it's no longer used. Even if you specify the stack name on the command, it still deploys all stacks it depends on. So I have to use the --exclusively flag.

cdk deploy --exclusively ecs-service
cdk deploy

In your case you would need a step before all that of creating the new cluster and deploying the stack so you have something new to import in ecs-service.

There is an open issue for this on GitHub.

I created the following script to help automate the process. It phases out the exports in two deployments. On the first deployment it restores any removed exports but marks them to be removed. This allows the first deployment to safely remove its usage. On the second deployment the script actually removes the exports after no other stack is using them.

To use the script you have to separate the synth and deploy steps and run the script in between them.

cdk synth && python phase-out-ref-exports.py && cdk deploy --app cdk.out --all

It requires permissions to read the stack, so it will probably not work well with cross-account deployments.

# phase-out-ref-exports.py
import json
import os
import os.path
 
from aws_cdk import cx_api
import boto3
import botocore.exceptions
 
 
def handle_template(stack_name, account, region, template_path):
    # get outputs from existing stack (if it exists)
    try:
        # TODO handle different accounts
        print(f"Checking exports of {stack_name}...")
        stack = boto3.client("cloudformation", region_name=region).describe_stacks(StackName=stack_name)["Stacks"][0]
        old_outputs = {
            o["OutputKey"]: o
            for o in stack.get("Outputs", [])
        }
    except botocore.exceptions.ClientError as e:
        print(f"Unable to phase out exports for {stack_name} on {account}/{region}: {e}")
        return
 
    # load new template generated by CDK
    new_template = json.load(open(template_path))
    if "Outputs" not in new_template:
        new_template["Outputs"] = {}
 
    # get output names for both existing and new templates
    new_output_names = set(new_template["Outputs"].keys())
    old_output_names = set(old_outputs.keys())
 
    # phase out outputs that are in old template but not in new template
    for output_to_phase_out in old_output_names - new_output_names:
        # if we already marked it for removal last deploy, remove the output
        if old_outputs[output_to_phase_out].get("Description") == "REMOVE ON NEXT DEPLOY":
            print(f"Removing {output_to_phase_out}")
            continue
 
        if not old_outputs[output_to_phase_out].get("ExportName"):
            print(f"This is an export with no name, ignoring {old_outputs[output_to_phase_out]}")
            continue
 
        # add back removed outputs
        print(f"Re-adding {output_to_phase_out}, but removing on next deploy")
        new_template["Outputs"][output_to_phase_out] = {
            "Value": old_outputs[output_to_phase_out]["OutputValue"],
            "Export": {
                "Name": old_outputs[output_to_phase_out]["ExportName"]
            },
            # mark for removal on next deploy
            "Description": "REMOVE ON NEXT DEPLOY",
        }
 
    # replace template
    json.dump(new_template, open(template_path, "w"), indent=4)
 
 
def handle_assembly(assembly):
    for s in assembly.stacks:
        handle_template(s.stack_name, s.environment.account, s.environment.region, s.template_full_path)
 
    for a in assembly.nested_assemblies:
        handle_assembly(a.nested_assembly)
 
 
def main():
    assembly = cx_api.CloudAssembly("cdk.out")
    handle_assembly(assembly)
 

if __name__ == "__main__":
    main()

I understood your issue like that: Create another cluster, migrate TaskDefinition with Service from old to the new cluster.

The thing is, that your old task is still is running as the error is telling you (SGs are still in use). Additionally, could it be the case that you are trying to re-use the security groups from the old cluster?

If not so, then you need to instantiate a new EcsServiceStack, but with the new cluster argument. Or if you don't care about a "manual blue/green deployment", then you can destroy the old EcsServiceStack. Then rerun the CDK commands with the modification of the code should run.

This was usefull for me. In my case I was using CDK and wanted to remove one stack (lets say stackA) with outputs referenced as input in another stacks (lets say stackB and stackC). So I broke all cross stack references puting the value manually on the cloudFormation template and then I deployed the template to every stack (stackB and stackC). Then I removed the stackA on CDK and deployed and it was successful.

If you are using AWS CDK for infra-as-code development, this blog CDK tips, part 3 – how to unblock cross-stack references written by Adam Ruka, who worked on CDK for almost 7 years in Amazon, could be a very good guidance.

If you are on CDK version 1.90.1 or later, the key is to use the exportValue function from the stack interface and do two consecutive deployments. The first one would ensure the parent stack preserves the exported values for the child stack to use.

See Removing automatic cross-stack references - GitHub aws-cdk-lib for more details.

That being said, I don't deny that this is a shortfall or design flaw that CDK and CloudFormation should have resolved more gracefully.

Another workaround is to manually create the outputs needed by setting the same exportName and value from the CloudFormation console in the cluster class:

new cdk.CfnOutput(this, `${this.id}clusterOutput`, { value: "sample-value", exportName: "ecs:ExportsOutputFnGetAttdefaultasgspotInstanceSecurityGroup2D2AFE98GroupId1084B7B2"});