What are best practices for minimizing CI Check times within a Code Repository?

Viewed 98

Are there things I can do to reduce the amount of time it takes for Code Repo checks to run?

2 Answers

If you're curious to get more detail about which step in your checks is taking the longest, you can add the following to your transforms-python/build.gradle file at the end:

import java.util.concurrent.TimeUnit
// Log timings per task.
class TimingsListener implements TaskExecutionListener, BuildListener {
    private long startTime
    private timings = []
    @Override
    void beforeExecute(Task task) {
        startTime = System.nanoTime()
    }
    @Override
    void afterExecute(Task task, TaskState taskState) {
        def ms = TimeUnit.MILLISECONDS.convert(System.nanoTime() - startTime, TimeUnit.NANOSECONDS);
        timings.add([ms, task.path])
        task.project.logger.warn "${task.path} took ${ms}ms"
    }
    @Override
    void buildFinished(BuildResult result) {
        println "Task timings:"
        for (timing in timings) {
            if (timing[0] >= 50) {
                printf "%7sms  %s\n", timing
            }
        }
    }
    @Override
    void buildStarted(Gradle gradle) {}
    @Override
    void projectsEvaluated(Gradle gradle) {}
    @Override
    void projectsLoaded(Gradle gradle) {}
    @Override
    void settingsEvaluated(Settings settings) {}
}
gradle.addListener new TimingsListener()

This can often reveal which step is the slowest, which when used in combination with the other answer can decrease your check times in a targeted fashion.

This is a very high level question, so I'm just dropping some bullet points as an answer, since there are so many things that can contribute to a long CI Check run. Once you make your question more deterministic I'll edit this.

Out of the top of my head here are the some common reasons why this happens:

  • You are importing a wide range of libraries, some with cross dependencies, bloating up your conda library resolution and making it take forever. A good resolution here would be to manually pin versions, to bypass the conda library resolution making it substancially faster.
  • Are you changing libraries to experiment with things? This will invalidate your conda cache forcing a cache rebuild which takes longer.
  • Are you on prem or within a Tennant with restricted internet access? Some steps of the CI download packages from the web, this can take time if your institution deployed foundry with slow internet access.
  • Your tests take a long time to run? Refactor your tests or optimize them somehow.
  • Are you in a monorepo? These often bloat in size and the CI takes longer because there is so much stuff in them.
Related