Can Jenkins build only one python package in a monorepo with multiple packages?

user_5615492 0
Viewed 1017

Suppose there is a monorepo with several independently buildable python packages in different directories, something the format shown below, where foo and bar are independently buildable.

Now suppose someone pushes code to a git repository where this is hosted, and the code revisions live within only the bar package. If Jenkins picks up the push, is there any way for Jenkins to detect which package was changed and only build and deploy the changed package, e.g. bar?

Or alternately, is it just standard practice to have Jenkins rebuild/redeploy all packages in a monorepo, regardless of what changes were pushed?.

root/
 └ src/
    ├ foo/
    │  ├ requirements.txt
    │  ├ setup.py
    │  └ foo/
    │     ├ __init__.py
    │     ├ module1.py
    │     ├ ...
    │     └ moduleN.py
    .
    .
    .
    └ bar/
       ├ requirements.txt
       ├ setup.py
       └ bar/
          ├ __init__.py
          ├ module1.py
          ├ ...
          └ moduleN.py
monorepo python-packaging jenkins packaging python
2 Answers

I have also been looking for this. I found a blog entry from Pinterest, where they use Pants:

https://medium.com/@Pinterest_Engineering/building-a-python-monorepo-for-fast-reliable-development-be763781f67

In one part they say:

Pants provides fast, reproducible builds for our packages. Pants performs incremental builds on its targets, so only changed modules are rebuilt which speeds up the build process. Running all the build targets in a virtual envs ensures builds are reproducible.

This looks good for me although Pants does not work on Windows.

user_108815660

I've been looking for the answer to this too. A manual job with a parameter to specify which package has changed is one way, but it's not great.

The other option is trying to detect what has changed, which I don't love either, but here's a git repo I found that has a script that does that https://github.com/slimm609/monorepo-gitwatcher

I'm hoping there is a better solution out there, I found a blog on Shazam (https://blog.shazam.com/python-microlibs-5be9461ad979) talking about it and it sounds like they have solved their problem. I've commented to try and get some clarification on that part.

I'm not excited about any of these solutions, but I can't think of or find anything better at the moment!

Edit: the response from Jorge on the Shazam article mentions they solve it by tagging the repo with a specific name and that indicates to Jenkins which of the packages to build in the monorepo. It's not completely automated, but I suppose it depends how much time you're willing to put into it vs how often it happens!

user_107103990
Related