CI failure in OSX x64 release NativeAOT SmokeTests SharedLibrary.sh

[carlossanlop]

See in a backport PR for release/7.0: #74894

Please determine if a fix needs to get backported to that branch.

• Queue: runtime (Build OSX x64 release NativeAOT)
• Job results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1712&view=logs&j=2de479a8-eae7-5324-c49d-bebaef329288&t=bb577e75-8247-55f1-4164-04f3b0f2d1eb&l=127
• Log message:

      nativeaot/SmokeTests/SharedLibrary/SharedLibrary/SharedLibrary.sh [FAIL]
        
        cmdLine:/Users/runner/work/1/s/artifacts/tests/coreclr/OSX.x64.Release/nativeaot/SmokeTests/SharedLibrary/SharedLibrary/SharedLibrary.sh Timed Out (timeout in milliseconds: 1800000 from variable __TestTimeout, start: 9/1/2022 9:28:03 PM, end: 9/1/2022 9:58:03 PM)
        
        Return code:      -100
        Raw output file:      /Users/runner/work/1/s/artifacts/tests/coreclr/OSX.x64.Release/nativeaot/SmokeTests/Reports/nativeaot.SmokeTests/SharedLibrary/SharedLibrary/SharedLibrary.output.txt
        Raw output:
        BEGIN EXECUTION
        /Users/runner/work/1/s/src/tests/Common/scripts/nativeaottest.sh /Users/runner/work/1/s/artifacts/tests/coreclr/OSX.x64.Release/nativeaot/SmokeTests/SharedLibrary/SharedLibrary/ SharedLibrary.dll ''
        
        cmdLine:/Users/runner/work/1/s/artifacts/tests/coreclr/OSX.x64.Release/nativeaot/SmokeTests/SharedLibrary/SharedLibrary/SharedLibrary.sh Timed Out (timeout in milliseconds: 1800000 from variable __TestTimeout, start: 9/1/2022 9:28:03 PM, end: 9/1/2022 9:58:03 PM)
        Test Harness Exitcode is : -100
        To run the test:
        > set CORE_ROOT=/Users/runner/work/1/s/artifacts/tests/coreclr/OSX.x64.Release/Tests/Core_Root
        > /Users/runner/work/1/s/artifacts/tests/coreclr/OSX.x64.Release/nativeaot/SmokeTests/SharedLibrary/SharedLibrary/SharedLibrary.sh
        Expected: True
        Actual:   False
        Stack Trace:
          /Users/runner/work/1/s/artifacts/tests/coreclr/OSX.x64.Release/TestWrappers/nativeaot.SmokeTests/nativeaot.SmokeTests.XUnitWrapper.cs(1634,0): at nativeaot_SmokeTests._SharedLibrary_SharedLibrary_SharedLibrary_._SharedLibrary_SharedLibrary_SharedLibrary_sh()
             at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
             at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
        Output:
          
          cmdLine:/Users/runner/work/1/s/artifacts/tests/coreclr/OSX.x64.Release/nativeaot/SmokeTests/SharedLibrary/SharedLibrary/SharedLibrary.sh Timed Out (timeout in milliseconds: 1800000 from variable __TestTimeout, start: 9/1/2022 9:28:03 PM, end: 9/1/2022 9:58:03 PM)
          
          Return code:      -100
          Raw output file:      /Users/runner/work/1/s/artifacts/tests/coreclr/OSX.x64.Release/nativeaot/SmokeTests/Reports/nativeaot.SmokeTests/SharedLibrary/SharedLibrary/SharedLibrary.output.txt
          Raw output:
          BEGIN EXECUTION
          /Users/runner/work/1/s/src/tests/Common/scripts/nativeaottest.sh /Users/runner/work/1/s/artifacts/tests/coreclr/OSX.x64.Release/nativeaot/SmokeTests/SharedLibrary/SharedLibrary/ SharedLibrary.dll ''
          
          cmdLine:/Users/runner/work/1/s/artifacts/tests/coreclr/OSX.x64.Release/nativeaot/SmokeTests/SharedLibrary/SharedLibrary/SharedLibrary.sh Timed Out (timeout in milliseconds: 1800000 from variable __TestTimeout, start: 9/1/2022 9:28:03 PM, end: 9/1/2022 9:58:03 PM)
          Test Harness Exitcode is : -100
          To run the test:
          > set CORE_ROOT=/Users/runner/work/1/s/artifacts/tests/coreclr/OSX.x64.Release/Tests/Core_Root
          > /Users/runner/work/1/s/artifacts/tests/coreclr/OSX.x64.Release/nativeaot/SmokeTests/SharedLibrary/SharedLibrary/SharedLibrary.sh
    Finished:    nativeaot.SmokeTests.XUnitWrapper
  === TEST EXECUTION SUMMARY ===
     nativeaot.SmokeTests.XUnitWrapper  Total: 17, Errors: 0, Failed: 1, Skipped: 0, Time: 1803.297s
/Users/runner/work/1/s/src/tests/Common/tests.targets(108,5): error MSB3073: The command ""/Users/runner/work/1/s/dotnet.sh" --roll-forward latestmajor /Users/runner/work/1/s/artifacts/tests/coreclr/OSX.x64.Release/Tests/Core_Root/xunit/xunit.console.dll /Users/runner/work/1/s/artifacts/tests/coreclr/OSX.x64.Release/nativeaot/SmokeTests/nativeaot.SmokeTests.XUnitWrapper.dll -parallel collections -html /Users/runner/work/1/s/artifacts/log/TestRun.html -xml /Users/runner/work/1/s/artifacts/log/TestRun.xml  -notrait category=outerloop -notrait category=failing -nocolor" exited with code 1. [/Users/runner/work/1/s/src/tests/build.proj]

Build FAILED.

/Users/runner/work/1/s/src/tests/Common/tests.targets(108,5): error MSB3073: The command ""/Users/runner/work/1/s/dotnet.sh" --roll-forward latestmajor /Users/runner/work/1/s/artifacts/tests/coreclr/OSX.x64.Release/Tests/Core_Root/xunit/xunit.console.dll /Users/runner/work/1/s/artifacts/tests/coreclr/OSX.x64.Release/nativeaot/SmokeTests/nativeaot.SmokeTests.XUnitWrapper.dll -parallel collections -html /Users/runner/work/1/s/artifacts/log/TestRun.html -xml /Users/runner/work/1/s/artifacts/log/TestRun.xml  -notrait category=outerloop -notrait category=failing -nocolor" exited with code 1. [/Users/runner/work/1/s/src/tests/build.proj]

Report

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	0

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[sbomer]

I put this in the 7.0 milestone - @MichalStrehovsky would you agree? Feel free to adjust.

[jkotas]

We have known crashes and hangs on osx-x64. #73299 is the top one.

osx-x64 is not officially supported in .NET 7. This does not need to be fixed in .NET 7.

We can disable this test on osx-x64 in servicing if it becomes a problem.

[carlossanlop]

For the record, I am seeing it again in another release/7.0 PR: #75042
Job results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=3827&view=logs&j=2de479a8-eae7-5324-c49d-bebaef329288&t=bb577e75-8247-55f1-4164-04f3b0f2d1eb

@jkotas @MichalStrehovsky if it's not officially supported, can you please disable the tests in 7.0? Feel free to reopen this issue so the PR resolves it.

[MichalStrehovsky]

osx-x64 is not officially supported in .NET 7. This does not need to be fixed in .NET 7.

This started happening around the time background GC was enabled. I wonder if background GC made things worse for macOS. We can disable the test for mac but I wonder if we should consider disabling background GC on mac as well given people would still like to try using it (#61231 (comment)). If the SharedLibrary test is crashing, the GC is probably completely unusable (the test really doesn't do much besides triggering a collection).

[VSadov]

When trying to build locally with ./build.sh -arch x64 -s OSX -s clr.aot+libs -rc Release -lc Release /p:CrossBuild=false I am geting errors like:

  /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: libcoreclrpal.a(context.S.o) has no symbols
  /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: libcoreclrpal.a(context.S.o) has no symbols

Is something missing in my local setup?
(I am able to do a regular clr build regular ARM64, but x64 has these issues)

[VSadov]

I also hit #74973 , which is probably causing the above error.
The x64 on M1 build appears to be broken.

[VSadov]

old x64 machine seems to be able to build, but it will take time

[VSadov]

I am able to reproduce the failure. Somehow bgc_start_event fails on OSX.
Wait returns a failure.

[MichalStrehovsky]

Fixed in #75298.