Jenkins Multibranch Pipeline check only the new or changed files pushed to GIT

Question

we are working with a tool where the elements are all written via .json.

Currently my GIT folder structure looks like this:

elementsfolder -> element1.json, element2.json, element3.json

scriptsfolder -> e.g. transformation.py

testfolder -> run-element.sh

Jenkinsfile

The problem I have now is that the .json files in folder elementsfolder should be tested with the bashscript from the testfolder and (if necessary) a script like transformations.py should be called. This also works so far, but all .json files are always tested (no matter if unchanged or not). But this should not be the case. Only the elements that are either changed or newly created should be tested. We have at the end of the day over 6000 elements, accordingly, the test over all elements would be too costly. Can anyone help me with this? In Jenkins the pipeline looks like this (I only post the stage build, because test and deploy are similar):

 stages {
        stage('Build') {
            environment {
                CREDS = credentials('creds')
                ENDPOINT = 'automation-api'
            }
            steps {
                sh '''
                username=$USR
                password=$PSW

                # Login curl
                login=$(curl -k -s -H "Content-Type: application/json" -X POST -d \{\"username\":\"$username\",\"password\":\"$password\"\} "$ENDPOINT/session/login" )
                token=$(echo ${login##*token\" : \"} | cut -d '"' -f 1)

                # Build
                #1. Validation .json
                curl -k -s -H "Authorization: Bearer $token" -X POST -F "definitionsFile=@element/*.json" "$ENDPOINT/build"
                #2. Show Error
                curl --show-error --fail -k -s -H "Authorization: Bearer $token" -X POST -F "definitionsFile=@element/*.json" "$ENDPOINT/build"
                #3. Logout
                curl -k -s -H "Authorization: Bearer $token" -X POST "$ENDPOINT/session/logout"
                '''
            }
        }

user_2627018 · Answer

If you are checking out from an SCM Jenkins will allow you to check the Changeset between the last build and the current build. So basically you can first try to find the files that really changed and then use only those files. Refer to the following example. After getting the list of files that changed you can either move them to a new directory or delete the unchanged files from the elementsfolder. The following sample getFilesChanged will return a list of all the changed/added files in the repo. You may want to tweak this function to match your requirement.

pipeline {
  agent any
  stages { 
      stage('clone') {
              steps {
                    git branch: 'main', url: 'https://github.com/xxx/sample.git'
              }
      }
      stage('build') {
              steps {
                  script {
                        println(getFilesChanged())
                        // Do your cleanup here and then execute the SH block
                  }
              }
      }
  }
}

def getFilesChanged() {
    def filesList = []
    def changeLogSets = currentBuild.changeSets
                    for (int i = 0; i < changeLogSets.size(); i++) {
                        def entries = changeLogSets[i].items
                        for (int j = 0; j < entries.length; j++) {
                            def entry = entries[j]
                            def files = new ArrayList(entry.affectedFiles)
                                for (int k = 0; k < files.size(); k++) {
                                def file = files[k]
                                filesList.add(file.path)
                }
            }
        }
    return filesList
}

Jenkins Multibranch Pipeline check only the new or changed files pushed to GIT

1 Answers