AWS CDK V2: CodePipeline + ShellStep + GitHub Integration

joker1979 0
Viewed 70

I am trying to use code pipeline and code build to create a cdk deployment where my github repo consists of several projects defined in a single github repo.

My Project Structure

Github:

/microservice1

/microservice2

Banging my head against this one for some time and reaching out to see if anyone else has experience this:

My CDK Code is like this:

    const pipeline = new CodePipeline(this, `${id}-code-pipline`, {        
    pipelineName: `${id}-code-pipline`,       
    role,                                    
    synth: new ShellStep(`${id}-code-pipeline-synth`, {
      input: CodePipelineSource.gitHub('joeMama123/stupid', 'master', {
        authentication:  SecretValue.secretsManager('stackoverflow', {
          jsonField: "WHY_AWS_DO_THIS_TO_ME_ON_EVERY_NEW_SERVICE_I_USE"
        }),            
      }),          
      env: {
        stage: "dev",
      },                    
      commands: [            
        'cd microservice1 && npm ci && npx cdk synth',
        'cd microservice2 && npm ci && npx cdk synth',                         
      ],
      primaryOutputDirectory: '', //have more than on location here.          
    }),
            
  });

The issue seems that the shell does not preserve the directory structure for subsequent commands. Its almost like the directory vanishes after the first command executes the command.

How can I preserve the initial github checkout structure so that I can navigate between the microservices, perform individual builds and synths and the deploy them from a single pipeline.

Any help would be greatly appreciated!!

aws-codebuild aws-codepipeline
1 Answers

I believe you are trying to use a tool that is designed for something else. CodePipeline is for deploying apps with CDK, it is not about building their sources.

What you are looking for is building your services with CodeBuild, create artifacts and deploy them with this pipeline.

Alternative solutions:

  • You could also add a deployment stage for an environment and add "pre" and "post" steps. In a "pre" step, you could have multiple ShellSteps to change directories and build individual apps. Creating an abstraction to handle this is also possible. If you have homogeneous apps, I would consider choosing this solution. Please note that its not possible to trigger individual "cdk synth" per "services". You will end up with a stage like "Deploy: Sandbox" with pre-steps building every application via Steps (ShellStep like: cd app/1 && npm run build:sandbox) and then your application's Stack would refer to the dist directory.

  • You can build the pipeline and grab its source action (I believe its pipeline.pipeline.actions...[0]), pick its inputSource, create a new CodeBuild project and add individual actions to build apps, where the inputSource of your first action is the previously mentioned one. Eg. You build a new CodeBuild project based on the source code that was grabbed by CodePipeline and you can modify, add stages to build individual dependencies. It is ugly solution, and it is a hack, you should not do this.

  • You can create a new abstraction that replaces your ShellStep that creates the output for synth. Check the underlying interface for implementation details. I would create it in a way that accepts individual app descriptors (eg new Application("microservice1")) as a property and would iterate through their directories, extracting build commands and running them before cdk synth.

What you should also keep in mind that if this synth step fails, cdk synth will not be completed and you might end up in a loop, even with self-mutating pipelines. Self-mutation happens after this step is done, til then, the previous version of the pipeline runs. This means that whatever triggers this pipeline, should be aware that it either modifies the pipeline's source, or the apps' build&deploy process.

If you would separate app builds and deployments to a different stage (pre shellsteps), it would fix the previous issue.

iMike0
Related