AWS CodePipeline track multiple repositories as a single source

user_1949177 0
Viewed 3199

I'm investigating whether or not CodePipeline will work for our use case:

We have several hundred repositories, all hosted with CodeCommit. The build/test/release process for all of these is identical, aside from minor configuration.

What I'd like to do is set up a pipeline that will build/test/release changes made in any of those repositories. I don't want to build all repositories, only the one that changed.

I'd rather not create 100 identical pipelines with different source repositories...

Is this possible? If not, are there any workarounds?

aws-codepipeline amazon-web-services
2 Answers

CodePipeline is designed around a model of one pipeline per project / service.

Trying to combine multiple independent release processes into a single pipeline is challenging due to the superseding behavior where newer changes can replace older changes while waiting for a stage to be come free to avoid unnecessary releases.

It will also make certain features such as the release history much less useful.

In your scenario, 100 pipelines is probably the best approach. CodePipeline recently raised the default limit of pipelines per account per region to 300 so you should have no problems in terms of limits.

You can use tools like CloudFormation to simplify setup and configuration of many similar pipelines. Also keep in mind that CodePipeline doesn't charge for pipelines which have no releases during a given month, so if you have many pipelines but you only actually make releases in a few of them per month then it won't cost you as much.

user_74331050

You may have to use a mix of CloudWatch rules + AWS Lambda + CodePipeline

CloudWatch Rules

  1. CodeCommit State Changes --> CloudWatch Rules --> Invoke Lambda (create a file which contains information with regard which repo and branch etc (ex: state-change.json). -> upload file to S3 bucket (ex: pipeline_source).

  2. S3 (pipeline_source) Events -> CloudWatch --> Trigger pipeline

CodePipeline

[ Source (S3 state-change.json, BuildSpec.yml, etc..)] --> [ Build (CodeBuild, custom script reads through the statechange.json file to determine what to build and deploy] -->
[ Deploy (...)]

Approach # 2 Parallel execution (different repos updated at the same time )

I don't think it's possible to wire up a single pipeline to meet your needs because of how pipeline runs the revisions. An alternative is to replace codePipeline with an step function state machine. It allows parallel execution. You can wireup the step function with actions that fits your CI/CD needs. I had a quick look at aws service integrations not many are supported. You need to use lambda function to invoke other aws services for ex: codebuild

CodeCommit state changes --> CloudWatch Rule --> Invoke Step function state machine

user_82475670
Related