share volume between agent block in jenkins k8s

user_3566441 0
Viewed 329

we have a jenkins pipeline that looks sort of like this

pipeline {
    agent {
        kubernetes {
            inheritFrom 'default'
            yamlFile 'pipeline_agent.yaml'
        }
    }
    stages {
        stage('build backend') { ... }
        stage('build frontend') { ... }
        stage('combine to jar file') { ... }
        stage('test') { ... }
    }
}

our pipeline_agent.yaml then contains containers for all different stages, so one for building with gradle, one with node for building the frontend, a separate custom one for testing, and so on.

And so this "works" and the output is great, but the issue is that we are provisioning a lot of resources that are not used for the entire duration on the pipeline.

For example it would be great to release all unused containers and then increase the resource requests for the test stage.

But I am not entirely sure how to do that. I can add an agent block in each stage and then I would be able to provision containers as needed, but then I don't think I can share data/build output between them.

Is there any "standard" or good way of doing this sort of dynamic provisioning?

jenkins-pipeline kubernetes jenkins
2 Answers

I think the best way is to have an agent block at each stage freeing up the containers after the job completes. As for sharing data between containers, you might want to use a shared persistent volume mounted in each container. This can be achieved by the NFS server provisioner

The NFS server provisioner would create Persistent Volume(Local one should work in your case) and an NFS storageclass. Your agent pods then can then mount and share the volume provisioned by this storageclass.

user_114579500
user_33150030
Related