Background
I have implemented a Swift algorithm in which I want to consume low memory at runtime (Space Complexity). So, I wanted to write performance tests using XCTMemoryMetric as part of a "Unit Testing Bundle" for large input sizes. I need to track regressions and optimisations for this algorithm, so I wanted initially to track the baseline metrics and expected this to work out of the box for my iOS app. Unfortunately, even after reading the Apple documentation for Performance Tests, I haven't found any potential fixes or information to help me resolve my particular issue.
I am using an iOS Base SDK for my main target (UIKit and Storyboard) and the same Base SDK for my "Unit Testing Bundle" in Xcode. In Xcode, I have left the default "Build active architecture only" setting to "Yes", and the main target compiles as well, and the tests compile fine. However, when I finish running the measure block with an XCTMemoryMetric() and XCTClockMetric() added as the only parameters, the Memory metrics don't show up in the test report. The Clock metrics are also completely unreliable, and my test fails (almost) every time due to high standard deviations even with the same function! Anyway, this question is focused on the memory metric.
Example of successful compilation:

Note: I have also tried other metrics like XCTCPUMetric for performance testing, which doesn't show results either.
Example of missing memory metrics:

I don't get warning messages in the console or exceptions.
What I've tried:
- Moving the algorithm to a
Swift Package Targetand moving performance tests to aPackage Unit Testing Bundle- the memory metric shows up in the test logs. Still, it is highly unreliable (sometimes showing even negative memory allocation !). This solution is not ideal since I don't want to create a package just for a single algorithm. - Moving the algorithm to a
Mac-only Targetwith correlatedUnit Testing Bundle- works but again not appropriate for an iOS-only app. - Debugging using
metric.reportMeasurements()shows a missing_dataparameter in the metric and crashes at runtime:
- Running individual test cases instead of the whole suite didn't make a difference.
- Testing even larger input array sizes didn't make a difference.
Is there any clean way I can resolve my issue without changing my Base SDK? I am considering abandoning XCTest for performance testing since it doesn't seem to work very well, but ideally, I'd like to stick with Apple's recommended APIs. You can also see the code seems to run completely on Stack Memory (no static lets or vars), so I don't think the confusion between Stack and Heap Memory is the root cause.
Code:
I've posted a gist of the algorithm I'm performance testing since the code is too many lines to include here. Even with other (less memory-intensive) code inside the measure block, the issue is reproducible for me.
The test code:
func testPerformanceExample() {
let metric = XCTMemoryMetric()
measure(metrics: [metric],
block: {
var sums: [[[Int]]] = []
(1...100).forEach { _ in
sums.append(QOneSolutionOne().fourSum([1, 0, -1, 0, -2, 2, 3, -3, 0, 4,
-2, 5, -4, 6, -6, 7, -8, 8, -7],
target: 0))
}
})
}
I think this may be an XCTest bug, but I wanted to see if I am missing something. Open to any advice, thanks.
