What tool or technique do you use to synchronize configuration across repositories?

user_2990349 0
Viewed 526

We have several base repositories that we use to quickly scaffold new projects. These include base repositories for things like:

  • TypeScript + Node libraries
  • TypeScript + Node microservices / workers
  • Angular applications
  • gRPC stub / protobuf libraries
  • etc.

The base repositories contain the all the configuration and folder structure necessary to start a new project, similar to ng new, create-react-app, yeoman generators, etc. This includes things like CI config, testing config, linting config, tsconfig, package.json with scripts and dev dependencies, VSCode files for settings, extension recommendations, launch configs, etc. Basically the repos contain everything you need to share between these similar repositories except the code.

There are tools out there for sharing code between repos, like bit, or you can publish libraries to share code as well, which we make extensive use of. However, I haven't found a great way to share the configuration files between many repositories. We could simply clone from base repositories or write our own generators, but the problem is keeping the configuration in sync. For instance, if we change our CI config, I don't want to have to remember which 50 repositories are based on that which also need changed.

We can't be the only firm out here with over 100 repos and no end in sight to the growth as we split into more and more distributed services and libraries. We can adopt the monorepository pattern, but it still doesn't change the fact that each of these projects inside the monorepo still needs its own config that has to be kept in sync across all of the projects.

So the question is, what tool or technique am I missing that can handle keeping these configurations in sync across many different repositories?

microservices config repository git
2 Answers

Here's how we currently do this:

  • create a new folder and run git init inside of it
  • add the remote (origin) for your repository
  • add a secondary remote (base) to the base repository
  • set the push url to an invalid URL so you can't push to the base repository (e.g. git remote set-url --push base NOPUSHALLOWED)
  • fetch the base repository
  • checkout the base's master branch into a local master branch with the --no-track flag to break the tracking link
  • replace some placeholders in the config files for your project (e.g. repo name in README.md, repo name in docker build npm script, etc.)
  • commit and push the local master to origin and set it to track origin's master

As part of our build scripts, before you can push a branch and create a PR, the script checks if this repository was created from a base repository (based on file artifacts in the repo) and if it was, it checks if there are changes in base that aren't present in your repo with git fetch base && git log --oneline --exit-code master..base/master > /dev/null. If there are any changes, your build will fail and it will prompt you to run a separate script to update from the base.

This has the benefit of making the synchronization automatic, because you can't make changes to a project without updating the configuration to the newest version, and the only time the config needs updated is when you're going to make changes and re-build.

On the other hand, it has downsides. It is annoying, to say the least, that you only find out you have to sync with the base after you've made your changes and run the build script. You could remember to run the check manually before you make changes, but let's be honest, that's not going to happen. Now you need to commit your changes, run the script to update from base, push and create a PR, wait for that to be merged, pull down a fresh master, and rebase/merge to update your branch - all of that before you can push your simple change. And that's a real pain when the change is critical and needs to go to production.

Another downside is that remotes are not synched across machines. So if I just git clone one of the applications, I don't have the base remote set up and so the build script will fail because it can't check against a remote it doesn't know about. We have solved this with another script in our internal CLI app that is used for cloning and sets up the base remote automatically, but I don't know of a way to allow developers to have write access to a repository but not be able to clone it (and only allow that programmatically) - not on GitHub, anyway.

I'm hoping someone has a better answer.

user_29903490

It's a cool problem to solve. I feel we have a pretty hacky solution to avoid duplicating work. We share eslintrc and prettierrc between repos in a folder that is included in .gitignore. The package.json script points to the config files in that folder.

It's not its own repo, and we use husky pre-commit hook to run the lint and format scripts. We don't run it when PRs are made (maybe we should). Right now there's nothing that stops you from using the no-verify flag to commit bad code, but code review should pick that up.

user_106194170
Related