Why do failing tests cause entire Jenkins build to be marked as "UNSTABLE" by Allure plugin?

Viewed 1409

I have set up Jenkins test jobs that will run Cucumber tests and want my jobs to always be successful up until a certain threshold (which I am controlling). The Allure plugin is always marking my builds as "UNSTABLE" when I have failed tests but I do not want my builds to be unstable. How can I change this behavior with allure?

I have tried setting <testFailureIgnore>true</testFailureIgnore> in pom.xml which is allowing the Maven Build to be successful. However, I am manually checking the Allure results in prometheusData.txt that is generated and failing/passing builds based off that result.

I want to be able to set Success/Failure status based off the 80% threshold. I am not aware of any thresholding that exists with Allure right now which is why I had to do it this way

Pipeline script

def testsInfo = readFile "${env.WORKSPACE}/allure-report/export/prometheusData.txt"

def passingTests = 0;
def totalTests = 0 
def lineText = ''
def lineNum = 0
while(lineText != 'found'){
    findText = testsInfo.split("\n")[lineNum]
    if(findText.contains('launch_status_passed')){
        passingTests = findText.split(' ')[1].toInteger()
    }
    if(findText.contains('launch_retries_run')){
        totalTests = findText.split(' ')[1].toInteger()
        lineText='found'
    }
    lineNum++
}

echo 'passing tests: ' + passingTests + '    total tests: ' + totalTests
percentSuccess = ((passingTests / totalTests) * 100).toString().split('\\.')[0]
echo 'Percent of tests that passed: ' + percentSuccess


if(build_ok && (percentSuccess.toInteger() > 80)) {
    currentBuild.result = 'SUCCESS'
    echo 'build OK'
} else {
    currentBuild.result = 'FAILURE'
    echo 'build bad'
}

Jenkins output

[INFO] 
[INFO] Results:
[INFO] 
[ERROR] Failures: 
[ERROR]   expected:<true> but was:<false>
[ERROR]   expected:<[1].00> but was:<[2].00>
[ERROR]   expected:<1.[0]0> but was:<1.[4]0>
[INFO] 
[ERROR] Tests run: 90, Failures: 3, Errors: 0, Skipped: 0
[INFO] 
[ERROR] There are test failures.
[INFO] -----------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] -----------------------------------------------------------------
[INFO] Total time: 04:05 min
[INFO] Finished at: 2019-06-24T17:31:36Z
[INFO] Final Memory: 33M/1349M
[INFO] -----------------------------------------------------------------
Allure report was successfully generated.
Creating artifact for the build.
Artifact was added to the build.
[Pipeline] // stage
[Pipeline] readFile
[Pipeline] echo
passing tests: 96    total tests: 100
[Pipeline] echo
Percent of tests that passed: 96
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: UNSTABLE
0 Answers
Related