Building a NET6/NET7 iOS project on agent M2 ARM 64 hangs/freezes

[rolfbjarne]

From @vincentcastagna on Thu, 16 Mar 2023 14:21:20 GMT

Steps to Reproduce

1. Create an agent on M2 ARM 64 agent (3.214.0)
2. Build a .NET6/.NET7 iOS project
3. Notice the build MIGHT hang sometimes on Apple Clang process

We don't face the issue on X64 on prem agents or even hosted.
There is no real consistency on when the build will hangs or not. It depends on the run.

We already tried removing Trimmer, which doesn't seem to have any effect. With or without, the behavior is the same.

Expected Behavior

Build should never hang

Actual Behavior

Build hangs sometimes and never ends, until timeout

Environment

• Xcode 14.2
• Visual Studio for mac 17.5.1
• This is the .csproj that we try to build

AGENT CAPABILITIES :

Agent.Name	MACOS-2C83F31C-42D1-4BA5-9686-611EB3632BD4
	Agent.Version	3.214.0
	_	./externals/node16/bin/node
	__CF_USER_TEXT_ENCODING	0x1F5:0x0:0x52
	CP_HOME_DIR	/Users/administrator/agent/_work/_temp/.cocoapods
	curl	/usr/bin/curl
	dotnet	/usr/local/share/dotnet/dotnet
	DOTNET_ROOT	/usr/local/share/dotnet
	git	/usr/bin/git
	HOME	/Users/administrator
	InteractiveSession	False
	java	/usr/bin/java
	JDK	/usr/bin/javac
	LANG	en_CA.UTF-8
	LOGNAME	administrator
	make	/usr/bin/make
	MSBuild	/Library/Frameworks/Mono.framework/Versions/Current/Commands/msbuild
	NUGET_HTTP_CACHE_PATH	/Users/administrator/agent/_work/_temp/.nuget-http-cache
	NUGET_PACKAGES	/Users/administrator/agent/_work/_temp/.nuget
	PATH	/Users/administrator/.rbenv/shims:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/share/dotnet:~/.dotnet/tools:/Library/Apple/usr/bin:/Library/Frameworks/Mono.framework/Versions/Current/Commands
	PWD	/Users/administrator/agent
	python3	/usr/bin/python3
	rake	/Users/administrator/.rbenv/shims/rake
	ruby	/Users/administrator/.rbenv/shims/ruby
	sh	/bin/sh
	SHELL	/bin/zsh
	SSH_AUTH_SOCK	/private/tmp/com.apple.launchd.MgBJHUlv5M/Listeners
	TMPDIR	/var/folders/33/ph0v51hd30n2frx557550mnc0000gn/T/
	USER	administrator
	VSTS_AGENT_SVC	1
	Xamarin.iOS	/Applications/Visual Studio.app/Contents/MacOS/vstool
	Xamarin.iOS_Version	16.1.1
	XamarinBuildDownloadDir	/Users/administrator/agent/_work/_temp/.xbcache
	xcode	/Applications/Xcode.app/Contents/Developer
	Xcode_Version	14.2
	XPC_FLAGS	0x0
	XPC_SERVICE_NAME	0

Build Logs

• Working build logs =>
  iOS BUILD - OK.txt

• Hanging build logs =>
  iOS BUILD - HANGS.txt

MSBUILD BINLOG (seem corrupted ...)

build-net7.0-ios.zip

Example Project (If Possible)

https://github.com/nventive/UnoApplicationTemplate/blob/dev/vica/make-usage-new-agents-net7.0/src/app/ApplicationTemplate.Mobile/ApplicationTemplate.Mobile.csproj

Copied from original issue xamarin/xamarin-macios#17825

[rolfbjarne]

From @rolfbjarne on Thu, 16 Mar 2023 14:40:58 GMT

@vincentcastagna have you ever seen this on an M1 machine? or have you never tried on M1?

[rolfbjarne]

From @msftbot[bot] on Thu, 16 Mar 2023 14:42:00 GMT

Hi @vincentcastagna. We have added the "need-info" label to this issue, which indicates that we have an open question for you before we can take further action. This issue will be closed automatically in 7 days if we do not hear back from you by then - please feel free to re-open it if you come back to this issue after that time.

[rolfbjarne]

From @vincentcastagna on Thu, 16 Mar 2023 15:34:27 GMT

@vincentcastagna have you ever seen this on an M1 machine? or have you never tried on M1?

I don't believe we tried on a M1 machine with a .NET6/.NET7 iOS project (only Xamarin.IOS). We will try asap and give feedback here.

[rolfbjarne]

From @vincentcastagna on Thu, 16 Mar 2023 17:27:08 GMT

I believe this might also be linked with this issue that I opened recently on ADO pipelines repo : microsoft/azure-pipelines-agent#4205

[rolfbjarne]

From @vincentcastagna on Mon, 20 Mar 2023 15:07:04 GMT

@rolfbjarne We have tested on a M1 machine. The behavior is exactly the same, sometimes it builds successfully, sometimes it just hangs. Seem to be happening half the time, exactly like M2.

iOS BUILD M1 - HANGS.txt

[rolfbjarne]

From @vincentcastagna on Wed, 22 Mar 2023 20:42:33 GMT

@rolfbjarne Not sure what extra info you would need ? I can provide.
mattjohnsonpint comments are unrelated to this issue I believe.

I omitted to precise that those agents are fully working on ANY Xamarin project, the build is super fast and never fails. Only our .NET6/.NET7 agents randomly hangs.

[rolfbjarne]

From @rolfbjarne on Thu, 23 Mar 2023 08:49:39 GMT

@vincentcastagna I'm assuming you only see this when building in Azure DevOps, and never locally?

One theory is that something pops up a permission dialog for some reason, and that blocks the build until it times out. Unfortunately these issues can be hard to track down unless you can access the built bot remotely (and catch it when the build is stuck).

One idea might be to make the build as verbose as possible, that should pinpoint a bit better exactly where it stops, and this is done by passing /v:diagnostic to the dotnet command:

dotnet build myapp.csproj /v:diagnostic

Could you do this and see what it shows?

[rolfbjarne]

From @vincentcastagna on Fri, 24 Mar 2023 17:26:00 GMT

@rolfbjarne here logs with /v:diagnostic you can see the instruction at the top of the logs. I don't see a real difference with or without this instruction. I have access to the machine of the agent, and I have never seen a permission dialog poping up though ... even in CLI logs or else.

iOS BUILD diagnostics - HANGS.txt

iOS BUILD diagnostics - OK.txt

[rolfbjarne]

From @rolfbjarne on Mon, 27 Mar 2023 15:35:15 GMT

I don't see a real difference with or without this instruction.

Because right after /v:diagnostic it's changed again to -verbosity:n:

/v:diagnostic -verbosity:n

[rolfbjarne]

From @vincentcastagna on Mon, 27 Mar 2023 17:24:31 GMT

I don't see a real difference with or without this instruction.

Because right after /v:diagnostic it's changed again to -verbosity:n:

/v:diagnostic -verbosity:n

Oh my bad, I missunderstood your previous comment, I will provide logs wtih verbosity level set to diagnostic asap.

[rolfbjarne]

From @vincentcastagna on Tue, 28 Mar 2023 19:46:38 GMT

@rolfbjarne plz find attached two logs diag level, one hanging, the other successful, needed to zip it as it exceeds file limitation of 25mb

LOGS AGENTS.zip

[rolfbjarne]

From @rolfbjarne on Wed, 29 Mar 2023 19:54:33 GMT

@vincentcastagna unfortunately that didn't give any new clues.

The next I'd try would be:

• Figure out exactly which process is hanging.
  • Activity Monitor can help here (is any process consuming 100% CPU?).
  • Alternatively use pstree [1], which will give you the entire process tree, and you can see where the build is stuck.
  • Using the Activity Monitor, it might also be useful to get a sample report (double-click the process, lower left corner there's a "Sample" button, which will sample the process for a few seconds and give a report about where the time was spent).
• Assuming it's the AOT compiler that gets stuck: is it always when compiling the same assembly? Or is it a different assembly on every build?

[1]: can be installed with brew install pstree (if you first install brew).

[rolfbjarne]

From @vincentcastagna on Fri, 31 Mar 2023 16:56:47 GMT

@rolfbjarne

So this is how the ActivityMonitor looks like after 45 min of hanging :

I have sampled multiple processes :

• Dotnet
  Sample of dotnet.txt

• Agent.Worker
  Sample of Agent.Worker.txt

• VisualStudio
  Sample of Visual Studio.txt

• The pstree output
  PstreeOutput.txt

• Yes we do compile the same assembly. But we also tested by compiling another set of assemblies, and the behavior is exactly the same.

[rolfbjarne]

From @rolfbjarne on Mon, 10 Apr 2023 14:18:06 GMT

The pstree info is potentially interesting, but unfortunately the important bits have been cut off:

|-+= 17099 administrator /bin/bash /Users/administrator/agent2/runsvc.sh
 | \-+- 17108 administrator ./externals/node16/bin/node ./bin/AgentService.js
 |   \-+- 17120 administrator /Users/administrator/agent2/bin/Agent.Listener ru
 |     \-+- 48081 administrator /Users/administrator/agent2/bin/Agent.Worker sp
 |       |--- 48093 administrator /Users/administrator/agent2/bin/Agent.PluginH
 |       \-+- 48804 administrator /Users/administrator/agent2/externals/node16/
 |         \-+- 48805 administrator /bin/bash /Users/administrator/agent2/_work
 |           \-+- 48806 administrator dotnet publish -f:net7.0-ios -c:Release -
 |             |--- 48807 administrator /Users/administrator/agent2/_work/_temp
 |             |--- 48808 administrator /Users/administrator/agent2/_work/_temp
 |             |--- 49623 administrator <defunct>
 |             |--- 49638 administrator <defunct>
 |             |--- 49648 administrator <defunct>
 |             |--- 49652 administrator <defunct>
 |             |--- 49654 administrator <defunct>
 |             |--- 49656 administrator <defunct>
 |             |--- 49658 administrator <defunct>
 |             |--- 49660 administrator <defunct>
 |             |--- 49662 administrator <defunct>
 |             |--- 49664 administrator <defunct>
 |             |--- 49665 administrator <defunct>
 |             \--- 49668 administrator <defunct>

the dotnet publish process is waiting for two other processes to finish, but the output doesn't say which processes those were, because the output was cut off at 80 characters. Could you try again, and somehow not truncate the output (my pstree doesn't do that, so I'm not sure how to fix it)?

[rolfbjarne]

From @vincentcastagna on Tue, 11 Apr 2023 17:00:34 GMT

@rolfbjarne I will provide a new pstree and checking that its not truncated. For now I'm kind of blocked due to this error : dotnet/installer#16038

[rolfbjarne]

From @vincentcastagna on Tue, 11 Apr 2023 18:54:21 GMT

@rolfbjarne well I believe its because I did not output to file directly.

Please find here the full pstree not truncated.

FullPstreeOutput.txt

[rolfbjarne]

From @rolfbjarne on Tue, 11 Apr 2023 19:08:14 GMT

I wonder if you're running into this: #6753

Can you try setting MSBUILDENSURESTDOUTFORTASKPROCESSES=1 in the environment to see if that changes something?

[rolfbjarne]

From @vincentcastagna on Tue, 11 Apr 2023 20:33:14 GMT

@rolfbjarne just to be sure setting MSBUILDENSURESTDOUTFORTASKPROCESSES=1 in the environment is passing this as an msbuild arguments /p:MSBUILDENSURESTDOUTFORTASKPROCESSES=1 ?
If that's the case, then it did not change the behavior. I can still post a new pstree output with this new arg.

Probably more like that, right ?

      env:
        MSBUILDENSURESTDOUTFORTASKPROCESSES: 1

[rolfbjarne]

From @rolfbjarne on Tue, 11 Apr 2023 20:43:47 GMT

@rolfbjarne just to be sure setting MSBUILDENSURESTDOUTFORTASKPROCESSES=1 in the environment is passing this as an msbuild arguments /p:MSBUILDENSURESTDOUTFORTASKPROCESSES=1 ? If that's the case, then it did not change the behavior. I can still post a new pstree output with this new arg.

Probably more like that, right ?

      env:
        MSBUILDENSURESTDOUTFORTASKPROCESSES: 1

Yes, like that.

[rolfbjarne]

From @vincentcastagna on Wed, 12 Apr 2023 15:18:15 GMT

Hey @rolfbjarne so I tried with both, as an msbuild arguments and set in the env, because I'm not sure the dotnet process will capture the env variable, so explicitly passing it probably would ensure that. But in any case, both being set, the behavior remains the same.
PsTreeWithMsbuildEnsureStdout.txt

[rolfbjarne]

From @rolfbjarne on Wed, 12 Apr 2023 15:26:35 GMT

@vincentcastagna can you try passing /nodeReuse:false to dotnet build as well?

dotnet build /nodeReuse:false ...

[rolfbjarne]

From @vincentcastagna on Wed, 12 Apr 2023 16:17:56 GMT

@vincentcastagna can you try passing /nodeReuse:false to dotnet build as well?

dotnet build /nodeReuse:false ...

I just passed the arguments to dotnet publish, still hanging. Here is pstree output.

PsTreeNodeReuse.txt

[rolfbjarne]

From @vincentcastagna on Wed, 19 Apr 2023 19:38:25 GMT

@rolfbjarne any news regarding this ? Can I provide more logs or else to help you investigate this matter.

[rolfbjarne]

From @rolfbjarne on Thu, 20 Apr 2023 16:11:56 GMT

@vincentcastagna I'm sorry I didn't answer earlier, but unfortunately I don't have any good ideas.

I see you're building the 'Release' configuration, does the same thing happen if you build 'Debug'? If so, one idea might be to turn off LLVM (by setting <MtouchUseLlvm>false</MtouchUseLlvm> in the project file or on the command line as /p:MtouchUseLlvm=false and see if that makes a difference).

[rolfbjarne]

From @vincentcastagna on Tue, 25 Apr 2023 15:22:17 GMT

@vincentcastagna I'm sorry I didn't answer earlier, but unfortunately I don't have any good ideas.

I see you're building the 'Release' configuration, does the same thing happen if you build 'Debug'? If so, one idea might be to turn off LLVM (by setting <MtouchUseLlvm>false</MtouchUseLlvm> in the project file or on the command line as /p:MtouchUseLlvm=false and see if that makes a difference).

We already tried deactivating LLVM when I created the issue, but in case, I retried. And the behavior remains the same, sometimes it goes through, sometimes it just hangs.

[rolfbjarne]

From @rolfbjarne on Wed, 26 Apr 2023 07:48:35 GMT

@vincentcastagna I'm sorry I didn't answer earlier, but unfortunately I don't have any good ideas.
I see you're building the 'Release' configuration, does the same thing happen if you build 'Debug'? If so, one idea might be to turn off LLVM (by setting <MtouchUseLlvm>false</MtouchUseLlvm> in the project file or on the command line as /p:MtouchUseLlvm=false and see if that makes a difference).

We already tried deactivating LLVM when I created the issue, but in case, I retried. And the behavior remains the same, sometimes it goes through, sometimes it just hangs.

What about a debug build that's not signed, so something like this (i.e. dotnet build instead of dotnet publish, and not passing /p:CodesignProvision=...)):

dotnet build -f:net7.0-ios ...

[rolfbjarne]

From @filipnavara on Fri, 28 Apr 2023 21:07:04 GMT

If you happen to have a way to run something on the machine with the stuck process then dotnet-stack would be useful (more info here). You install the tool with dotnet tool install --global dotnet-stack and then run it with dotnet stack report -p <id of the stuck process>. Something like pgrep dotnet | xargs -L1 dotnet stack report -p would dump stacks of all the dotnet processes on the machine.

[rolfbjarne]

From @vincentcastagna on Tue, 23 May 2023 18:04:41 GMT

I have ran a dotnet stack report -p for each msbuild processes I found running using pstree once a build hangs. I don't see much information here, but hopefully this would be useful to you :

msbuildstack.zip

@filipnavara I tried to run pgrep dotnet | xargs -L1 dotnet stack report -p , but the command line gets frozen and nothing happens. I also tried to write the output in a file but it just hangs

[rolfbjarne]

From @filipnavara on Tue, 23 May 2023 18:54:25 GMT

Both of the stack traces contain OutOfProcNode.Run so they seem to be waiting for some other MSBuild (?) process.

I tried to run pgrep dotnet | xargs -L1 dotnet stack report -p , but the command line gets frozen and nothing happens.

There are two possible explanations for this. Either I messed up and it's trying to dump itself in a loop, or some process is stuck so badly that not even the diagnostic pipes work. The former is not very likely since I tested that very same command locally. The later would likely imply hitting some .NET runtime bug (and there's only one thread-suspension bug that comes to mind which was fixed in .NET 7 iirc)...

[rolfbjarne]

From @vincentcastagna on Tue, 23 May 2023 19:42:50 GMT

Thank you for your quick answer.

Both of the stack traces contain OutOfProcNode.Run so they seem to be waiting for some other MSBuild (?) process.

As you saw I found two msbuild processes ... could it be that they wait on each other, driving an endless waiting loop. Any advice maybe to try confirm that or seek for other processes that would be waited by msbuild ?

I decided to let pgrep dotnet | xargs -L1 dotnet stack report -p run . Finally ended ...

[ERROR] System.IO.EndOfStreamException: Unable to read beyond the end of the stream.
   at System.IO.BinaryReader.InternalRead(Int32 numBytes)
   at System.IO.BinaryReader.ReadUInt16()
   at Microsoft.Diagnostics.NETCore.Client.IpcHeader.Parse(BinaryReader[ERROR] System.IO.EndOfStreamException: Unable to read beyond the end of the stream.
   at System.IO.BinaryReader.InternalRead(Int32 numBytes)
   at System.IO.BinaryReader.ReadUInt16()
   at Microsoft.Diagnostics.NETCore.Client.IpcHeader.Parse(BinaryReader reader) in /_/src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsIpc/IpcHeader.cs:line 55
   at Microsoft.Diagnostics.NETCore.Client.IpcMessage.Parse(Stream stream) in /_/src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsIpc/IpcMessage.cs:line 117
   at Microsoft.Diagnostics.NETCore.Client.IpcClient.Read(Stream stream) in /_/src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsIpc/IpcClient.cs:line 107
   at Microsoft.Diagnostics.NETCore.Client.IpcClient.SendMessageGetContinuation(IpcEndpoint endpoint, I reader) in /_/src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsIpc/IpcHeader.cs:line 55
   at Microsoft.Diagnostics.NETCore.Client.IpcMessage.Parse(Stream stream) in /_/src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsIpc/IpcMessage.cs:line 117
   apcMessage message) in /_/src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsIpc/IpcClient.cs:line 44
   at Microsoft.Diagnostics.NETCore.Client.EventPipeSession.Start(IpcEndpoint endpoint, IEnumerable`1 providers, Boolean requestRundown, Int32 circularBufferMB) in /_/src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsClient/EventPipeSession.cs:line 34
   at Microsoft.Diagnostics.Tools.Stack.ReportCommandHandler.Report(CancellationToken ct, IConsole console, Int32 processId, String name, TimeSpan duration)t Microsoft.Diagnostics.NETCore.Client.IpcClient.Read(Stream stream) in /_/src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsIpc/IpcClient.cs:line 107
   at Microsoft.Diagnostics.NETCore.Client.IpcClient.SendMessageGetContinuation(IpcEndpoint endpoint, I
pcMessage message) in /_/src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsIpc/IpcClient.cs:line 44
   at Microsoft.Diagnostics.NETCore.Client.EventPipeSession.Start(IpcEndpoint endpoint, IEnumerable`1 providers, Boolean requestRundown, Int32 circularBufferMB) in /_/src/Microsoft.Diagnostics.NETCore.Client/DiagnosticsClient/EventPipeSession.cs:line 34
   at Microsoft.Diagnostics.Tools.Stack.ReportCommandHandler.Report(CancellationToken ct, IConsole console, Int32 processId, String name, TimeSpan duration)
xargs: dotnet: exited with status 255; aborting

I'll also try to target latest .NET 7

[rolfbjarne]

From @svaldetero on Tue, 27 Jun 2023 20:51:13 GMT

I think I'm running into this issue also. I recently moved from microsoft hosted to a self-hosted M2 Max MacStudio. Changing nothing in the pipeline definition, the command line dotnet publish 'ProjectName.csproj' -f net7.0-ios --self-contained -r ios-arm64 -c Release -p:BuildIpa=True always freezes and eventually times out at 60 minutes or I have to cancel it. I tried switching it to dotnet build 'ProjectName.csproj' -f net7.0-ios -c Release and it has the same result. What's frustrating is I can copy the exact command to terminal and run it in the same directory and it works just fine.

I tried running dotnet stack but it just hung and never finished. I got the same EndOfStreamException when I finally cancelled the pipeline.

[rolfbjarne]

From @rolfbjarne on Wed, 28 Jun 2023 09:04:51 GMT

At this point I believe this is either a bug in msbuild or in the runtime, not in any of our MSBuild logic, so I'm moving to dotnet/msbuild.

[rokonec]

We, me and @AR-May, have thoroughly investigated it and it seems that it is hanging in Task
https://github.com/xamarin/xamarin-macios/blob/61493dd43817b97700bd92d90b958e30688b8457/msbuild/Xamarin.MacDev.Tasks/Tasks/CompileNativeCode.cs#L8 . By comparing OK and Failed logs it looks like this task failed before it started all clang processes.

I have also notice that, if I read this code correctly, implementation of
https://github.com/xamarin/xamarin-macios/blob/61493dd43817b97700bd92d90b958e30688b8457/msbuild/Xamarin.MacDev.Tasks/Tasks/CompileNativeCodeTaskBase.cs#L99
and https://github.com/xamarin/xamarin-macios/blob/61493dd43817b97700bd92d90b958e30688b8457/msbuild/Xamarin.MacDev.Tasks/Tasks/AOTCompileTaskBase.cs#L215 and other tasks which using ExecuteAsync + WaitAll approach, will create and run given tool, like clang, in new Thread per each assembly.
In context of this particular issue, it will run 150 concurrent clang processes unbounded and all those concurrent processes will fight for shared resources like CPU, memory and disk IO.
In theory it should not be the reason for hangs, but this could result in performance degradation and massive system overbooking with increasing probability of dead locks, live locks and other concurrency issues/bugs.

@rolfbjarne please, if you deem our analysis correct, close this issue and reopen the original one.

[rolfbjarne]

@rokonec thanks a lot for the analysis!

That certainly sounds like something that could cause problems, and would be a good fix even if it turns out to not be the actual issue, so I'll go ahead and reopen the other issue.