The OSX 12.00 iPhone queue stopped showing iPhones

[premun]

This started sometime last week and 100% of the runs now end with that we fail tu run mlaunch (MacOS app) and the invoked mlaunch process hangs and doesn't return an exit code:

[04:33:54] info: Preparing run for ios-device
[04:33:54] info: Looking for available ios-device devices..
[04:33:54] dbug: Looking for available ios-device devices. Storing logs into list-ios-device-20220913_043354.log
[04:36:15] crit: System.Exception: Failed to list devices.
                    at Microsoft.DotNet.XHarness.iOS.Shared.Hardware.HardwareDeviceLoader.LoadDevices(ILog log, Boolean includeLocked, Boolean forceRefresh, Boolean listExtraData, Boolean includeWirelessDevices, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.XHarness.iOS.Shared/Hardware/HardwareDeviceLoader.cs:line 103
                    at Microsoft.DotNet.XHarness.Apple.DeviceFinder.FindDevice(TestTargetOs target, String deviceName, ILog log, Boolean includeWirelessDevices, Boolean pairedDevicesOnly, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.XHarness.Apple/DeviceFinder.cs:line 74
                    at Microsoft.DotNet.XHarness.Apple.BaseOrchestrator.OrchestrateOperationInternal(TestTargetOs target, String deviceName, Boolean includeWirelessDevices, Boolean resetSimulator, Boolean enableLldb, GetAppBundleInfoFunc getAppBundle, ExecuteMacCatalystAppFunc executeMacCatalystApp, ExecuteAppFunc executeApp, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.XHarness.Apple/Orchestration/BaseOrchestrator.cs:line 215
                    at Microsoft.DotNet.XHarness.Apple.BaseOrchestrator.OrchestrateOperation(TestTargetOs target, String deviceName, Boolean includeWirelessDevices, Boolean resetSimulator, Boolean enableLldb, GetAppBundleInfoFunc getAppBundle, ExecuteMacCatalystAppFunc executeMacCatalystApp, ExecuteAppFunc executeApp, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.XHarness.Apple/Orchestration/BaseOrchestrator.cs:line 96
                    at Microsoft.DotNet.XHarness.Apple.RunOrchestrator.OrchestrateRun(GetAppBundleInfoFunc getAppBundleInfo, TestTargetOs target, String deviceName, TimeSpan timeout, TimeSpan launchTimeout, Int32 expectedExitCode, Boolean includeWirelessDevices, Boolean resetSimulator, Boolean enableLldb, Boolean signalAppEnd, Boolean waitForExit, IReadOnlyCollection`1 environmentalVariables, IEnumerable`1 passthroughArguments, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.XHarness.Apple/Orchestration/RunOrchestrator.cs:line 221
                    at Microsoft.DotNet.XHarness.CLI.Commands.Apple.AppleAppCommand`1.Invoke(ILogger logger) in /_/src/Microsoft.DotNet.XHarness.CLI/Commands/Apple/AppleAppCommand.cs:line 68
                    at Microsoft.DotNet.XHarness.CLI.Commands.XHarnessCommand`1.Invoke(IEnumerable`1 arguments) in /_/src/Microsoft.DotNet.XHarness.CLI/Commands/XHarnessCommand.cs:line 145
XHarness exit code: 71 (GENERAL_FAILURE)

This means we fail to list the devices:
https://github.com/dotnet/xharness/blob/55e8420fb913c035522bbb30e28d4fd7181ecd69/src/Microsoft.DotNet.XHarness.iOS.Shared/Hardware/HardwareDeviceLoader.cs#L103

https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-xharness-refs-pull-946-merge-aae83de0c49c4082a8/ios-device-iOS.Simulator.PInvoke.Test.app/1/console.350748e0.log?helixlogtype=result

[premun]

When I checked the other files (namely list-ios-device-20220913_043354.log) on the failing Helix job, I see this:

[04:33:54.4115520] 
[04:33:54.4117600] Running /private/tmp/helix/working/B89909A1/p/microsoft.dotnet.xharness.cli/1.0.0-ci/tools/net7.0/any/../../../runtimes/any/native/mlaunch/bin/mlaunch --sdkroot /Applications/Xcode_13.4.app --listdev=/tmp/helix/working/B89909A1/t/tmp7hBqv0.tmp --list-wireless-devices=false --output-format=XML --timeout=0.5 -v -v -v -v -v
[04:33:54.4289630] Launching /private/tmp/helix/working/B89909A1/p/microsoft.dotnet.xharness.cli/1.0.0-ci/tools/net7.0/any/../../../runtimes/any/native/mlaunch/bin/mlaunch --sdkroot /Applications/Xcode_13.4.app --listdev=/tmp/helix/working/B89909A1/t/tmp7hBqv0.tmp --list-wireless-devices=false --output-format=XML --timeout=0.5 -v -v -v -v -v
[04:35:54.4300330] Process 19849 didn't exit within 00:02:00 and will be killed
[04:35:54.4328330] Killing process tree of 19849...
[04:35:54.4680040] Pids to kill: 19849
[04:35:54.4681330] Running lldb diagnostics for pid 19849
[04:35:54.4691300] Printing backtrace for pid=19849
[04:35:54.5454430] (lldb) command source -s 0 '/tmp/helix/working/B89909A1/t/tmpbu5hh7.tmp'
[04:35:54.5456960] Executing commands in '/tmp/helix/working/B89909A1/t/tmpbu5hh7.tmp'.
[04:35:54.5457640] (lldb) process attach --pid 19849
[04:36:14.6481890] 19849 Execution timed out after 120 seconds and the process was killed.
[04:36:15.7705790] Failed to get ExitCode: System.InvalidOperationException: Process must exit before requested information can be determined.
   at System.Diagnostics.Process.EnsureState(State state)
   at System.Diagnostics.Process.get_ExitCode()
   at Microsoft.DotNet.XHarness.Common.Execution.ProcessManager.RunAsyncInternal(Process process, ILog log, ILog stdout, ILog stderr, Action`2 kill, Func`3 getChildProcessIds, Nullable`1 timeout, Dictionary`2 environmentVariables, Nullable`1 cancellationToken, Nullable`1 diagnostics) in /_/src/Microsoft.DotNet.XHarness.Common/Execution/ProcessManager.cs:line 347
[04:36:15.7722290] Failed to parse device list: System.Exception: Failed to list devices.
   at Microsoft.DotNet.XHarness.iOS.Shared.Hardware.HardwareDeviceLoader.LoadDevices(ILog log, Boolean includeLocked, Boolean forceRefresh, Boolean listExtraData, Boolean includeWirelessDevices, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.XHarness.iOS.Shared/Hardware/HardwareDeviceLoader.cs:line 103

[premun]

Things to try:

1. Reboot the machines in the queue
2. Try this out manually:
   1. Log into the machine over KVM / SSH
   2. Download XHarness nupkg and extract it
   3. Inside the nupkg is mlaunch (the app failing)
   4. Try the same command as is in the logs:

      microsoft.dotnet.xharness.cli/1.0.0-ci/runtimes/any/native/mlaunch/bin/mlaunch --sdkroot /Applications/Xcode_13.4.app --listdev=/tmp/helix/working/B89909A1/t/tmp7hBqv0.tmp --list-wireless-devices=false --output-format=XML --timeout=0.5 -v -v -v -v -v

3. Check what changes were done to the queue lately (MacOS updates?)

[oleksandr-didyk]

Closing as the issue seems to be resolved via steps described here