Our fastlane Xcode builds are failing intermittently when run by Jenkins. About 10% of builds, while running test targets, we'll get a failure with:
[2020-11-16T16:12:10.276Z] 2020-11-16 08:11:47.435 xcodebuild[72478:2549553] [MT] IDETestOperationsObserverDebug: 415.317 elapsed -- Testing started completed.
[2020-11-16T16:12:10.276Z] 2020-11-16 08:11:47.435 xcodebuild[72478:2549553] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start
[2020-11-16T16:12:10.276Z] 2020-11-16 08:11:47.435 xcodebuild[72478:2549553] [MT] IDETestOperationsObserverDebug: 415.317 sec, +415.317 sec -- end
[2020-11-16T16:12:10.276Z] Testing failed:
[2020-11-16T16:12:10.276Z] REDACTED_iPhoneTests:
[2020-11-16T16:12:10.276Z] REDACTED.app encountered an error (Failed to install or launch the test runner. If you believe this error represents a bug, please attach the result bundle at /tmp/build/jenkins-mobile-ios-develop-1107/ios/fastlane/output/derivedData/Logs/Test/Test-iPhoneSnapshotTests-2020.11.16_08-04-51--0800.xcresult. (Underlying Error: The request to open "com.REDACTED.REDACTEDapp" failed. The request was denied by service delegate (SBMainWorkspace) for reason: Busy ("Application cannot be launched because it has outstanding termination assertions"). (Underlying Error: The operation couldn’t be completed. Application cannot be launched because it has outstanding termination assertions.)))
In the simulator device logs we have a lot of "failed to bootstrap path" errors, where the report indicates that some path under Xcode-12.app cannot be found (even though in reality, it's there):
Failed to bootstrap path: path = /Applications/Xcode-12.app/Contents/Developer/Platforms/iPhoneOS.platform/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS.simruntime/Contents/Resources/RuntimeRoot/usr/lib/dyld_sim, error = 2: No such file or directory
Nov 16 08:11:44 Mac-169 com.apple.CoreSimulator.SimDevice.4A33E878-D3F6-4EE4-AB53-7ACD59E0B8A5[73117] (com.apple.xpc.launchd.domain.pid.testmanagerd.73149): Failed to bootstrap path: path = /Applications/Xcode-12.app/Contents/Developer/Platforms/iPhoneOS.platform/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS.simruntime/Contents/Resources/RuntimeRoot/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit, error = 2: No such file or directory
Nov 16 08:11:44 Mac-169 com.apple.CoreSimulator.SimDevice.4A33E878-D3F6-4EE4-AB53-7ACD59E0B8A5[73117] (com.apple.xpc.launchd.domain.pid.LogArchiveCollector.73942): Failed to bootstrap path: path = /Applications/Xcode-12.app/Contents/Developer/Platforms/iPhoneOS.platform/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS.simruntime/Contents/Resources/RuntimeRoot/usr/lib/dyld_sim, error = 2: No such file or directory
Nov 16 08:11:44 Mac-169 com.apple.CoreSimulator.SimDevice.4A33E878-D3F6-4EE4-AB53-7ACD59E0B8A5[73117] (com.apple.xpc.launchd.domain.pid.LogArchiveCollector.73942): Failed to bootstrap path: path = /Applications/Xcode-12.app/Contents/Developer/Platforms/iPhoneOS.platform/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS.simruntime/Contents/Resources/RuntimeRoot/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit, error = 2: No such file or directory
It seems that one of the simulator clones keeps crashing and relaunching possibly (this is from the same log as the first log above, but happens just before it. (I have removed the timestamps and inserted some separators to make it easier to read, and put this at the bottom of this post since it's the longest one):
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
xcodebuild IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
/tmp/build/jenkins-mobile-ios-develop-1107/ios/fastlane/output/derivedData/Logs/Test/Test-iPhoneSnapshotTests-2020.11.16_08-04-51--0800.xcresult/Staging/1_Test/Diagnostics/DETCADERSnapshotTests-BA58BA45-D47B-418B-8E61-80EBA8B3FAAF/DETCADERSnapshotTests-A6366A3A-D533-4145-A1A2-C487CD9787E8/Session-DETCADERSnapshotTests-2020-11-16_081142-AyG7h3.log
xcodebuild IDETestOperationsObserverDebug: (02A95C0D-288A-4F0B-BE2B-BB376A8C5A16) Beginning test session DETCADERSnapshotTests-02A95C0D-288A-4F0B-BE2B-BB376A8C5A16 at 2020-11-16 08:11:42.263 with Xcode 12A7209 on target <DVTiPhoneSimulator: 0x7fd44999f1e0> {
SimDevice: Clone 2 of iPhone 7 (8FEB1993-97B1-470B-9511-C5B4693A2634, iOS 14.0, Booted)
}
xcodebuild IDETestOperationsObserverDebug: (02A95C0D-288A-4F0B-BE2B-BB376A8C5A16) Finished requesting crash reports. Continuing with testing.
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
xcodebuild IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
/tmp/build/jenkins-mobile-ios-develop-1107/ios/fastlane/output/derivedData/Logs/Test/Test-iPhoneSnapshotTests-2020.11.16_08-04-51--0800.xcresult/Staging/1_Test/Diagnostics/DETCADERSnapshotTests-BA58BA45-D47B-418B-8E61-80EBA8B3FAAF/DETCADERSnapshotTests-DF0A8969-D65B-4379-8DD4-1C445437C40C/Session-DETCADERSnapshotTests-2020-11-16_081142-ojHiX7.log
xcodebuild IDETestOperationsObserverDebug: (B3B8C894-1650-4C81-955D-EB3A844A1796) Beginning test session DETCADERSnapshotTests-B3B8C894-1650-4C81-955D-EB3A844A1796 at 2020-11-16 08:11:42.522 with Xcode 12A7209 on target <DVTiPhoneSimulator: 0x7fd44999f1e0> {
SimDevice: Clone 2 of iPhone 7 (8FEB1993-97B1-470B-9511-C5B4693A2634, iOS 14.0, Booted)
}
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
xcodebuild IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
/tmp/build/jenkins-mobile-ios-develop-1107/ios/fastlane/output/derivedData/Logs/Test/Test-iPhoneSnapshotTests-2020.11.16_08-04-51--0800.xcresult/Staging/1_Test/Diagnostics/DETCADERSnapshotTests-BA58BA45-D47B-418B-8E61-80EBA8B3FAAF/DETCADERSnapshotTests-30F99A37-EB4E-4E36-9581-0EB2288A3664/Session-DETCADERSnapshotTests-2020-11-16_081142-RaEqDv.log
xcodebuild IDETestOperationsObserverDebug: (60E66AC3-343C-44E6-A0F1-CF1C4404CE76) Beginning test session DETCADERSnapshotTests-60E66AC3-343C-44E6-A0F1-CF1C4404CE76 at 2020-11-16 08:11:42.526 with Xcode 12A7209 on target <DVTiPhoneSimulator: 0x7fd44999f1e0> {
SimDevice: Clone 2 of iPhone 7 (8FEB1993-97B1-470B-9511-C5B4693A2634, iOS 14.0, Booted)
}
xcodebuild IDETestOperationsObserverDebug: (B3B8C894-1650-4C81-955D-EB3A844A1796) Finished requesting crash reports. Continuing with testing.
xcodebuild IDETestOperationsObserverDebug: (60E66AC3-343C-44E6-A0F1-CF1C4404CE76) Finished requesting crash reports. Continuing with testing.
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
xcodebuild IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
/tmp/build/jenkins-mobile-ios-develop-1107/ios/fastlane/output/derivedData/Logs/Test/Test-iPhoneSnapshotTests-2020.11.16_08-04-51--0800.xcresult/Staging/1_Test/Diagnostics/REDACTEDSnapshotTests-5A572528-16A8-4271-BEC6-5143DCFD3B6C/REDACTEDSnapshotTests-A02288EC-AA49-46A6-8760-231AD5913E59/Session-REDACTEDSnapshotTests-2020-11-16_081142-90pgsf.log
xcodebuild IDETestOperationsObserverDebug: (EE062564-9EDD-4C4A-9A21-77212982CCA8) Beginning test session REDACTEDSnapshotTests-EE062564-9EDD-4C4A-9A21-77212982CCA8 at 2020-11-16 08:11:42.913 with Xcode 12A7209 on target <DVTiPhoneSimulator: 0x7fd44999f1e0> {
SimDevice: Clone 2 of iPhone 7 (8FEB1993-97B1-470B-9511-C5B4693A2634, iOS 14.0, Booted)
}
xcodebuild IDETestOperationsObserverDebug: (EE062564-9EDD-4C4A-9A21-77212982CCA8) Finished requesting crash reports. Continuing with testing.
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
xcodebuild IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
/tmp/build/jenkins-mobile-ios-develop-1107/ios/fastlane/output/derivedData/Logs/Test/Test-iPhoneSnapshotTests-2020.11.16_08-04-51--0800.xcresult/Staging/1_Test/Diagnostics/REDACTEDSnapshotTests-5A572528-16A8-4271-BEC6-5143DCFD3B6C/REDACTEDSnapshotTests-25FCC3D3-3D12-41AE-8D6C-D8662913F0E4/Session-REDACTEDSnapshotTests-2020-11-16_081143-8gwjmz.log
xcodebuild IDETestOperationsObserverDebug: (696DAA83-6A4E-44BB-B341-E9419DC49450) Beginning test session REDACTEDSnapshotTests-696DAA83-6A4E-44BB-B341-E9419DC49450 at 2020-11-16 08:11:43.178 with Xcode 12A7209 on target <DVTiPhoneSimulator: 0x7fd44999f1e0> {
SimDevice: Clone 2 of iPhone 7 (8FEB1993-97B1-470B-9511-C5B4693A2634, iOS 14.0, Booted)
}
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
xcodebuild IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
/tmp/build/jenkins-mobile-ios-develop-1107/ios/fastlane/output/derivedData/Logs/Test/Test-iPhoneSnapshotTests-2020.11.16_08-04-51--0800.xcresult/Staging/1_Test/Diagnostics/REDACTEDSnapshotTests-5A572528-16A8-4271-BEC6-5143DCFD3B6C/REDACTEDSnapshotTests-E0063304-AC2D-4C96-A8F3-B5139FD2C72B/Session-REDACTEDSnapshotTests-2020-11-16_081143-CSWZve.log
xcodebuild IDETestOperationsObserverDebug: (D2EB889B-C942-4E3C-81FB-597689EB9B89) Beginning test session REDACTEDSnapshotTests-D2EB889B-C942-4E3C-81FB-597689EB9B89 at 2020-11-16 08:11:43.181 with Xcode 12A7209 on target <DVTiPhoneSimulator: 0x7fd44999f1e0> {
SimDevice: Clone 2 of iPhone 7 (8FEB1993-97B1-470B-9511-C5B4693A2634, iOS 14.0, Booted)
}
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
xcodebuild IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
/tmp/build/jenkins-mobile-ios-develop-1107/ios/fastlane/output/derivedData/Logs/Test/Test-iPhoneSnapshotTests-2020.11.16_08-04-51--0800.xcresult/Staging/1_Test/Diagnostics/REDACTEDSnapshotTests-5A572528-16A8-4271-BEC6-5143DCFD3B6C/REDACTEDSnapshotTests-1CA2FE03-1508-4A05-BC8F-742DA4396EAD/Session-REDACTEDSnapshotTests-2020-11-16_081143-k91bVJ.log
xcodebuild IDETestOperationsObserverDebug: (0B3DA48C-F46C-4B3E-87C0-2F688C1225D3) Beginning test session REDACTEDSnapshotTests-0B3DA48C-F46C-4B3E-87C0-2F688C1225D3 at 2020-11-16 08:11:43.185 with Xcode 12A7209 on target <DVTiPhoneSimulator: 0x7fd44999f1e0> {
SimDevice: Clone 2 of iPhone 7 (8FEB1993-97B1-470B-9511-C5B4693A2634, iOS 14.0, Booted)
}
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
xcodebuild IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
/tmp/build/jenkins-mobile-ios-develop-1107/ios/fastlane/output/derivedData/Logs/Test/Test-iPhoneSnapshotTests-2020.11.16_08-04-51--0800.xcresult/Staging/1_Test/Diagnostics/REDACTEDSnapshotTests-5A572528-16A8-4271-BEC6-5143DCFD3B6C/REDACTEDSnapshotTests-E715EB70-2866-42C7-A1DD-25224AEC982F/Session-REDACTEDSnapshotTests-2020-11-16_081143-kiKHeq.log
xcodebuild IDETestOperationsObserverDebug: (BD78FA29-C83A-4F69-BF73-37946869A4C4) Beginning test session REDACTEDSnapshotTests-BD78FA29-C83A-4F69-BF73-37946869A4C4 at 2020-11-16 08:11:43.189 with Xcode 12A7209 on target <DVTiPhoneSimulator: 0x7fd44999f1e0> {
SimDevice: Clone 2 of iPhone 7 (8FEB1993-97B1-470B-9511-C5B4693A2634, iOS 14.0, Booted)
Here is the associated crash log for testmanagerd:
Process: testmanagerd [72950]
Path: /Applications/Xcode-12.app/Contents/Developer/Platforms/iPhoneOS.platform/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS.simruntime/Contents/Resources/RuntimeRoot/usr/libexec/testmanagerd
Identifier: testmanagerd
Version: 17155
Code Type: X86-64 (Native)
Parent Process: launchd_sim [72917]
Responsible: SimulatorTrampoline [12382]
User ID: 501
Date/Time: 2020-11-16 08:12:07.926 -0800
OS Version: Mac OS X 10.15.7 (19H15)
Report Version: 12
Anonymous UUID: 5CA872E8-AE72-5688-477B-B3E543033B38
Time Awake Since Boot: 310000 seconds
System Integrity Protection: enabled
Crashed Thread: 0 Dispatch queue: com.apple.main-thread
Exception Type: EXC_CRASH (SIGKILL)
Exception Codes: 0x0000000000000000, 0x0000000000000000
Exception Note: EXC_CORPSE_NOTIFY
Termination Reason: Namespace LIBXPC, Code 0x4
Application Specific Information:
CoreSimulator 732.17 - Device: Clone 3 of iPhone 7 (229EEFDE-E207-4409-AAC8-B00A256DC6D1) - Runtime: iOS 14.0 (18A372) - DeviceType: iPhone 7
Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0 libsystem_kernel.dylib 0x00007fff5dc9fdfa mach_msg_trap + 10
1 libsystem_kernel.dylib 0x00007fff5dca0170 mach_msg + 60
2 com.apple.CoreFoundation 0x00007fff203a7e77 __CFRunLoopServiceMachPort + 316
3 com.apple.CoreFoundation 0x00007fff203a258d __CFRunLoopRun + 1284
4 com.apple.CoreFoundation 0x00007fff203a1b9e CFRunLoopRunSpecific + 567
5 com.apple.CoreFoundation 0x00007fff203a2cf1 CFRunLoopRun + 40
6 testmanagerd 0x000000010bcd8180 -[XCTDaemon run] + 399
7 testmanagerd 0x000000010bcb14fa main + 318
8 libdyld.dylib 0x00007fff20257415 start + 1
Thread 1:
0 libsystem_pthread.dylib 0x00007fff5dcd7b68 start_wqthread + 0
Thread 2:
0 libsystem_pthread.dylib 0x00007fff5dcd7b68 start_wqthread + 0
Thread 3:
0 libsystem_pthread.dylib 0x00007fff5dcd7b68 start_wqthread + 0
Thread 0 crashed with X86 Thread State (64-bit):
rax: 0x0000000010004005 rbx: 0x0000000007000806 rcx: 0x00007ffee3f4e638 rdx: 0x0000000000000000
rdi: 0x00007ffee3f4e820 rsi: 0x0000000007000806 rbp: 0x00007ffee3f4e690 rsp: 0x00007ffee3f4e638
r8: 0x0000000000001d03 r9: 0x00000000ffffffff r10: 0x0000000000000c00 r11: 0x0000000000000206
r12: 0x0000000007000806 r13: 0x0000000000000c00 r14: 0x00007ffee3f4e820 r15: 0x0000000000000000
rip: 0x00007fff5dc9fdfa rfl: 0x0000000000000206 cr2: 0x00007fff207357fa
Logical CPU: 0
Error Code: 0x0100001f
Trap Number: 133