AWS Cloudformation stuck in UPDATE_ROLLBACK_FAILED

Viewed 1313

I deploy my AWS Lambdas via AWS Serverless Application Model (SAM). One of my Lambdas uses Numpy which I reference via a 3rd party layer from Klayers by @keithRozario. I was using Klayers-python38-numpy:16 but discovered that it was deprecated after I deployed today which left my stack in an UPDATE_ROLLBACK_FAILED state.

One recommendation is to use Stack actions -> Continue update rollback from the AWS console; which I tried but it didn't work. The other solution is to delete the stack. However, this would be my first time deleting a stack and what I'd like to know is: if I delete my stack via the console, will my stack get recreated when I redeploy it? I've looked for answers to my question but I'm only finding references to deleting resources within the stack.

What I'd also like to know is, my stack is the first stack of many in an AWS CodePipeline, will my pipeline still work if I delete my stack? Further, will I experience anymore failed stacks as I proceed to subsequent stacks within my pipeline?

Lastly, the plan is to update to Klayers-python38-numpy:19 when I redeploy.

EDIT: as per @marcin

The problem is that the Klayers-python38-numpy:16, that is already deployed throughout my stack, is no longer available. I tried deploying a change to my code this morning, my pipeline failed during the CreateChangeSet step. The fact that this layer is no longer available is, I'm assuming, the reason my stack is unable to rollback.

My pipeline looks like this:


{
    "pipeline": {
        "name": "my-pipeline",
        "roleArn": "arn:aws:iam::123456789:role/my-pipeline-CodePipelineExecutionRole-4O8PAUJGLXYZ",
        "artifactStore": {
            "type": "S3",
            "location": "my-pipeline-buildartifactsbucket-62byf2xqaa8z"
        },
        "stages": [
            {
                "name": "Source",
                "actions": [
                    {
                        "name": "SourceCodeRepo",
                        "actionTypeId": {
                            "category": "Source",
                            "owner": "ThirdParty",
                            "provider": "GitHub",
                            "version": "1"
                        },
                        "runOrder": 1,
                        "configuration": {
                            "Branch": "master",
                            "OAuthToken": "****",
                            "Owner": "hugo",
                            "Repo": "my-pipeline"
                        },
                        "outputArtifacts": [
                            {
                                "name": "SourceCodeAsZip"
                            }
                        ],
                        "inputArtifacts": []
                    }
                ]
            },
            {
                "name": "Build",
                "actions": [
                    {
                        "name": "CodeBuild",
                        "actionTypeId": {
                            "category": "Build",
                            "owner": "AWS",
                            "provider": "CodeBuild",
                            "version": "1"
                        },
                        "runOrder": 1,
                        "configuration": {
                            "ProjectName": "my-pipeline"
                        },
                        "outputArtifacts": [
                            {
                                "name": "BuildArtifactAsZip"
                            }
                        ],
                        "inputArtifacts": [
                            {
                                "name": "SourceCodeAsZip"
                            }
                        ]
                    }
                ]
            },
            {
                "name": "CI",
                "actions": [
                    {
                        "name": "CreateChangeSet",
                        "actionTypeId": {
                            "category": "Deploy",
                            "owner": "AWS",
                            "provider": "CloudFormation",
                            "version": "1"
                        },
                        "runOrder": 1,
                        "configuration": {
                            "ActionMode": "CHANGE_SET_REPLACE",
                            "Capabilities": "CAPABILITY_IAM",
                            "ChangeSetName": "my-pipeline-ChangeSet-ci",
                            "ParameterOverrides": "{\n  \"MyEnvironment\" : \"ci\"\n}\n",
                            "RoleArn": "arn:aws:iam::123456789:role/my-pipeline-CloudFormationExecutionRole-1O8GOB5C2VXYZ",
                            "StackName": "my-pipeline-ci",
                            "TemplatePath": "BuildArtifactAsZip::packaged.yaml"
                        },
                        "outputArtifacts": [],
                        "inputArtifacts": [
                            {
                                "name": "BuildArtifactAsZip"
                            }
                        ]
                    },
                    {
                        "name": "ExecuteChangeSet",
                        "actionTypeId": {
                            "category": "Deploy",
                            "owner": "AWS",
                            "provider": "CloudFormation",
                            "version": "1"
                        },
                        "runOrder": 2,
                        "configuration": {
                            "ActionMode": "CHANGE_SET_EXECUTE",
                            "ChangeSetName": "my-pipeline-ChangeSet-ci",
                            "RoleArn": "arn:aws:iam::123456789:role/my-pipeline-CloudFormationExecutionRole-1O8GOB5C2VXYZ",
                            "StackName": "my-pipeline-ci"
                        },
                        "outputArtifacts": [
                            {
                                "name": "my-pipelineCIChangeSet"
                            }
                        ],
                        "inputArtifacts": []
                    }
                ]
            },
            {
                "name": "Staging",
                "actions": [
                    {
                        "name": "CreateChangeSet",
                        "actionTypeId": {
                            "category": "Deploy",
                            "owner": "AWS",
                            "provider": "CloudFormation",
                            "version": "1"
                        },
                        "runOrder": 1,
                        "configuration": {
                            "ActionMode": "CHANGE_SET_REPLACE",
                            "Capabilities": "CAPABILITY_IAM",
                            "ChangeSetName": "my-pipeline-ChangeSet-staging",
                            "ParameterOverrides": "{\n  \"MyEnvironment\" : \"staging\"\n}\n",
                            "RoleArn": "arn:aws:iam::123456789:role/my-pipeline-CloudFormationExecutionRole-1O8GOB5C2VXYZ",
                            "StackName": "my-pipeline-staging",
                            "TemplatePath": "BuildArtifactAsZip::packaged.yaml"
                        },
                        "outputArtifacts": [],
                        "inputArtifacts": [
                            {
                                "name": "BuildArtifactAsZip"
                            }
                        ]
                    },
                    {
                        "name": "ExecuteChangeSet",
                        "actionTypeId": {
                            "category": "Deploy",
                            "owner": "AWS",
                            "provider": "CloudFormation",
                            "version": "1"
                        },
                        "runOrder": 2,
                        "configuration": {
                            "ActionMode": "CHANGE_SET_EXECUTE",
                            "ChangeSetName": "my-pipeline-ChangeSet-staging",
                            "RoleArn": "arn:aws:iam::123456789:role/my-pipeline-CloudFormationExecutionRole-1O8GOB5C2VXYZ",
                            "StackName": "my-pipeline-staging"
                        },
                        "outputArtifacts": [
                            {
                                "name": "my-pipelineStagingChangeSet"
                            }
                        ],
                        "inputArtifacts": []
                    }
                ]
            },
            {
                "name": "Prod",
                "actions": [
                    {
                        "name": "DeploymentApproval",
                        "actionTypeId": {
                            "category": "Approval",
                            "owner": "AWS",
                            "provider": "Manual",
                            "version": "1"
                        },
                        "runOrder": 1,
                        "configuration": {},
                        "outputArtifacts": [],
                        "inputArtifacts": []
                    },
                    {
                        "name": "CreateChangeSet",
                        "actionTypeId": {
                            "category": "Deploy",
                            "owner": "AWS",
                            "provider": "CloudFormation",
                            "version": "1"
                        },
                        "runOrder": 2,
                        "configuration": {
                            "ActionMode": "CHANGE_SET_REPLACE",
                            "Capabilities": "CAPABILITY_IAM",
                            "ChangeSetName": "my-pipeline-ChangeSet-prod",
                            "ParameterOverrides": "{\n  \"MyEnvironment\" : \"prod\"\n}\n",
                            "RoleArn": "arn:aws:iam::123456789:role/my-pipeline-CloudFormationExecutionRole-1O8GOB5C2VXYZ",
                            "StackName": "my-pipeline-prod",
                            "TemplatePath": "BuildArtifactAsZip::packaged.yaml"
                        },
                        "outputArtifacts": [],
                        "inputArtifacts": [
                            {
                                "name": "BuildArtifactAsZip"
                            }
                        ]
                    },
                    {
                        "name": "ExecuteChangeSet",
                        "actionTypeId": {
                            "category": "Deploy",
                            "owner": "AWS",
                            "provider": "CloudFormation",
                            "version": "1"
                        },
                        "runOrder": 3,
                        "configuration": {
                            "ActionMode": "CHANGE_SET_EXECUTE",
                            "ChangeSetName": "my-pipeline-ChangeSet-prod",
                            "RoleArn": "arn:aws:iam::123456789:role/my-pipeline-CloudFormationExecutionRole-1O8GOB5C2VXYZ",
                            "StackName": "my-pipeline-prod"
                        },
                        "outputArtifacts": [
                            {
                                "name": "my-pipelineProdChangeSet"
                            }
                        ],
                        "inputArtifacts": []
                    }
                ]
            }
        ],
        "version": 1
    }
}
1 Answers

if I delete my stack via the console, will my stack get recreated when I redeploy it?

Yes. You can try to deploy same stack again, but probably you should investigate why it failed in the first place.

What I'd also like to know is, my stack is the first stack of many in an AWS CodePipeline, will my pipeline still work if I delete my stack?

Don't know, but probably not. Its use case specific and you haven't provide any info about the CP.

Further, will I experience anymore failed stacks as I proceed to subsequent stacks within my pipeline?

If one action fails, you can't proceed with further actions. Even if you could, other stacks can depend on the first one, and they will fail as well.

Related