I'm trying to setup a build pipeline with a test step, and want to include a basic performance test using measure(). When I run this test with a low baseline inside Xcode, it fails as it should. However, when I run the same tests using xcodebuild or within a service like Codemagic, the tests always pass because it doesn't seem to apply the baseline.
Is this even possible to do: run automated performance tests in a CI service using baselines?
I see this output:
Test Case '-[CodemagicDemoUITests.CodemagicDemoUITests testLaunchPerformance]' measured [Duration (AppLaunch), s] average: 0.456, relative standard deviation: 22.633%, values: [0.367859, 0.488878, 0.639593, 0.424477, 0.357795], performanceMetricID:com.apple.dt.XCTMetric_ApplicationLaunch-AppLaunch.duration, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.000, maxStandardDeviation: 0.000
"baselineAverage" is blank and there is no baseline name.