I'm dealing with a build that takes a long time to compile.
I came up with an unorthodox idea to speed up the build by leveraging the docker cache. This would be employed in a CI context where we are trying to (as quickly as possible of course) automatically produce a pass/fail on whether the code compiles, smoke/unit/integration test results, etc.
Traditionally we would run a build with a
COPY src src
RUN mkdir build && cmake .. && make -j$(nproc) && make install && cd .. && rm -rf build
The drawback with this is that we have to clean-build fresh even if we change something trivial in the project. Usually this is desired and I would continue to implement it this way for CD.
The first attempt to describe what we want to do is something along the lines of
MKDIR build
COPY src/util src/util
WORKDIR build
RUN cmake ..
RUN make -j$(nproc) util
# This is pseudocode
WORKDIR ..
COPY src/app1 src/app1
WORKDIR build
RUN cmake ..
RUN make -j$(nproc) app1
# etc.
Of course this incremental approach has many drawbacks (polluting layers etc) but that can be mitigated (multistage etc).
In theory what this lets us do is if only code in app1 changed then app1 would get built fresh but the util library files could be carried over from Docker cache!
Now the practical problem with this approach is that it's really just not going to work. Something like cmake was not built for this. The CMakeLists.txt is likely going to reference app1 inside of it and the very first cmake call will fail unless I do some really gnarly munging of the file to comment out the inclusion of app1 and all other dependents for that initial build of util.
So here is where my idea comes in. I'm thinking of basically doing a sort of "incremental" but not-quite-as-dirty as a typical incremental build.
- (This part kind of hand wavey) determine a good (recent-ish) commit from which to use the full source code for the next step. This will not be the actively worked-on code. Maybe we manually update this commit as we move along (so that this scheme can work at all under the scenarios involving actual cmake-related arch changes)
- Check out that commit from git in the dockerfile (docker build)
RUNthecmakecall on this static commitCOPYoverutilcodemake utilCOPYoverapp1codemake app1... etc.
I think the biggst gaping hole in this right now is the added burden of keeping track of some kind of hand-specified "cached" cmake seed commit. If this isn't recent enough then the build could fail due to a mismatch in the build files. Every time that commit gets updated it does force a full build at that point.
Unfortunately this seems like a scheme that is potentially too error-prone in practice to be worth pursuing. So I guess I shot my strategy down by myself already.
What might be another approach to this? I feel like there has to be some kind of way to do this or maybe some tool that is not even affiliated with Docker that can facilitate this concept of like flexibly caching an actual clean build.
For example it's clear to me that multistage dockerfiles can be used to do something flexible, for example we would have a stage for util, and sibling children of that stage would be app1, app2, etc. This would at least in theory allow for the possibility of only having to rebuild the app1 stage if only content in src/app1 had changed.
I think the problem I have is with how to represent or cue up the starting state from which we can build app1 and only app1 if we just change app1.
The obvious answer is to rewrite the build definition to make it so that it is possible to build util when only having access to source for util. So this ties back to the point above about munging CMakeLists.txt on the fly; could just write CMakeLists in a flexible enough way to allow this kind of incremental use. This is a bit unpalatable but probably doable.