Exception unwinding broken with Xcode 15 linker #92297

vcsjones · 2023-09-19T19:04:26Z

Recently I've started observing a strange behavior when attempting to run tests where first chance exceptions seem to be crashing the test runner.

For example, running: ./dotnet.sh test src/libraries/System.Formats.Cbor/tests

Produces:

The active test run was aborted. Reason: Test host process crashed : Unhandled exception. System.Formats.Cbor.CborContentException: The current CBOR map contains an incomplete key/value pair.
   at System.Formats.Cbor.CborReader.PeekStateCore() in /Users/vcsjones/Projects/runtime/src/libraries/System.Formats.Cbor/src/System/Formats/Cbor/Reader/CborReader.PeekState.cs:line 82
   at System.Formats.Cbor.CborReader.SkipNextNode(Int32& depth) in /Users/vcsjones/Projects/runtime/src/libraries/System.Formats.Cbor/src/System/Formats/Cbor/Reader/CborReader.SkipValue.cs:line 70
   at System.Formats.Cbor.CborReader.SkipToAncestor(Int32 depth, Boolean disableConformanceModeChecks) in /Users/vcsjones/Projects/runtime/src/libraries/System.Formats.Cbor/src/System/Formats/Cbor/Reader/CborReader.SkipValue.cs:line 51

I don't see any reason this exception should crash the test runner.

It's not just in S.F.Cbor. For Tar:

 Skipped System.Formats.Tar.Tests.TarWriter_WriteEntry_File_Tests.CreateEntryFromFileOwnedByNonExistentUser [1 ms]
The active test run was aborted. Reason: Test host process crashed : Unhandled exception. Unhandled exception. System.IO.IOException: The file '/var/folders/gc/qfr6cnsj3p1fc1944dkn1t6h0000gn/T/33hxgrfd.1wv/file.tar' already exists.
   at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirError) in /Users/vcsjones/Projects/runtime/src/libraries/Common/src/Interop/Unix/Interop.IOErrors.cs:line 16

I have my suspicion that the issue is due to upgrade Xcode to Xcode 15 (However I am on macOS 13 still).

Currently, I can't run any tests.

./dotnet.sh --info
.NET SDK:
 Version:   8.0.100-rc.1.23415.11
 Commit:    12f1505f6e

Runtime Environment:
 OS Name:     Mac OS X
 OS Version:  13.5
 OS Platform: Darwin
 RID:         osx-arm64
 Base Path:   /Users/vcsjones/Projects/runtime/.dotnet/sdk/8.0.100-rc.1.23415.11/

The text was updated successfully, but these errors were encountered:

ghost · 2023-09-19T19:04:41Z

Tagging subscribers to this area: @dotnet/area-system-formats-cbor, @bartonjs, @vcsjones
See info in area-owners.md if you want to be subscribed.

Issue Details

Recently I've started observing a strange behavior when attempting to run tests where first chance exceptions seem to be crashing the test runner.

For example, running: ./dotnet.sh test src/libraries/System.Formats.Cbor/tests

Produces:

The active test run was aborted. Reason: Test host process crashed : Unhandled exception. System.Formats.Cbor.CborContentException: The current CBOR map contains an incomplete key/value pair.
   at System.Formats.Cbor.CborReader.PeekStateCore() in /Users/vcsjones/Projects/runtime/src/libraries/System.Formats.Cbor/src/System/Formats/Cbor/Reader/CborReader.PeekState.cs:line 82
   at System.Formats.Cbor.CborReader.SkipNextNode(Int32& depth) in /Users/vcsjones/Projects/runtime/src/libraries/System.Formats.Cbor/src/System/Formats/Cbor/Reader/CborReader.SkipValue.cs:line 70
   at System.Formats.Cbor.CborReader.SkipToAncestor(Int32 depth, Boolean disableConformanceModeChecks) in /Users/vcsjones/Projects/runtime/src/libraries/System.Formats.Cbor/src/System/Formats/Cbor/Reader/CborReader.SkipValue.cs:line 51

I don't see any reason this exception should crash the test runner.

It's not just in S.F.Cbor. For Tar:

 Skipped System.Formats.Tar.Tests.TarWriter_WriteEntry_File_Tests.CreateEntryFromFileOwnedByNonExistentUser [1 ms]
The active test run was aborted. Reason: Test host process crashed : Unhandled exception. Unhandled exception. System.IO.IOException: The file '/var/folders/gc/qfr6cnsj3p1fc1944dkn1t6h0000gn/T/33hxgrfd.1wv/file.tar' already exists.
   at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirError) in /Users/vcsjones/Projects/runtime/src/libraries/Common/src/Interop/Unix/Interop.IOErrors.cs:line 16

I have my suspicion that the issue is due to upgrade Xcode to Xcode 15 (However I am on macOS 13 still).

Currently, I can't run any tests.

./dotnet.sh --info
.NET SDK:
 Version:   8.0.100-rc.1.23415.11
 Commit:    12f1505f6e

Runtime Environment:
 OS Name:     Mac OS X
 OS Version:  13.5
 OS Platform: Darwin
 RID:         osx-arm64
 Base Path:   /Users/vcsjones/Projects/runtime/.dotnet/sdk/8.0.100-rc.1.23415.11/

Author:	vcsjones
Assignees:	-
Labels:	`untriaged`, `area-System.Formats.Cbor`
Milestone:	-

ghost · 2023-09-19T19:05:10Z

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

Recently I've started observing a strange behavior when attempting to run tests where first chance exceptions seem to be crashing the test runner.

For example, running: ./dotnet.sh test src/libraries/System.Formats.Cbor/tests

Produces:

The active test run was aborted. Reason: Test host process crashed : Unhandled exception. System.Formats.Cbor.CborContentException: The current CBOR map contains an incomplete key/value pair.
   at System.Formats.Cbor.CborReader.PeekStateCore() in /Users/vcsjones/Projects/runtime/src/libraries/System.Formats.Cbor/src/System/Formats/Cbor/Reader/CborReader.PeekState.cs:line 82
   at System.Formats.Cbor.CborReader.SkipNextNode(Int32& depth) in /Users/vcsjones/Projects/runtime/src/libraries/System.Formats.Cbor/src/System/Formats/Cbor/Reader/CborReader.SkipValue.cs:line 70
   at System.Formats.Cbor.CborReader.SkipToAncestor(Int32 depth, Boolean disableConformanceModeChecks) in /Users/vcsjones/Projects/runtime/src/libraries/System.Formats.Cbor/src/System/Formats/Cbor/Reader/CborReader.SkipValue.cs:line 51

I don't see any reason this exception should crash the test runner.

It's not just in S.F.Cbor. For Tar:

 Skipped System.Formats.Tar.Tests.TarWriter_WriteEntry_File_Tests.CreateEntryFromFileOwnedByNonExistentUser [1 ms]
The active test run was aborted. Reason: Test host process crashed : Unhandled exception. Unhandled exception. System.IO.IOException: The file '/var/folders/gc/qfr6cnsj3p1fc1944dkn1t6h0000gn/T/33hxgrfd.1wv/file.tar' already exists.
   at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirError) in /Users/vcsjones/Projects/runtime/src/libraries/Common/src/Interop/Unix/Interop.IOErrors.cs:line 16

I have my suspicion that the issue is due to upgrade Xcode to Xcode 15 (However I am on macOS 13 still).

Currently, I can't run any tests.

./dotnet.sh --info
.NET SDK:
 Version:   8.0.100-rc.1.23415.11
 Commit:    12f1505f6e

Runtime Environment:
 OS Name:     Mac OS X
 OS Version:  13.5
 OS Platform: Darwin
 RID:         osx-arm64
 Base Path:   /Users/vcsjones/Projects/runtime/.dotnet/sdk/8.0.100-rc.1.23415.11/

Author:	vcsjones
Assignees:	-
Labels:	`os-mac-os-x`, `area-Infrastructure`, `untriaged`
Milestone:	-

adamsitnik · 2023-09-20T06:21:40Z

cc @janvorli

vitek-karas · 2023-09-20T18:13:13Z

@MarcoRossignoli FYI

vcsjones · 2023-09-20T22:07:36Z

This is what I was able to gather with lldb.

frame #0: 0x00000001a14f4764 libsystem_kernel.dylib`__pthread_kill + 8
    frame #1: 0x00000001a152bc28 libsystem_pthread.dylib`pthread_kill + 288
    frame #2: 0x00000001a1439ae8 libsystem_c.dylib`abort + 180
    frame #3: 0x00000001030e84ec libcoreclr.dylib`PROCAbort(signal=6, siginfo=0x0000000000000000) at process.cpp:2558:5
    frame #4: 0x00000001030e8080 libcoreclr.dylib`PROCEndProcess(hProcess=0x00000000ffffff01, uExitCode=1, bTerminateUnconditionally=YES) at process.cpp:1352:13
    frame #5: 0x00000001030e818c libcoreclr.dylib`TerminateProcess(hProcess=0x00000000ffffff01, uExitCode=1) at process.cpp:1249:11
    frame #6: 0x00000001031f09a0 libcoreclr.dylib`CrashDumpAndTerminateProcess(exitCode=1) at excep.cpp:4220:5
    frame #7: 0x00000001035d3bd0 libcoreclr.dylib`UnwindManagedExceptionPass1(ex=0x0000000170558d18, frameContext=0x0000000170558820) at exceptionhandling.cpp:4877:17
    frame #8: 0x00000001035d40e8 libcoreclr.dylib`DispatchManagedException(ex=0x0000000170558d18, isHardwareException=false) at exceptionhandling.cpp:4956:17
    frame #9: 0x0000000103480908 libcoreclr.dylib`IL_Rethrow() at jithelpers.cpp:4365:5

Frame 7 looks interesting. If I switch to that controlPc is 0.

I'm not sure what to make of this, but I'm more or less at a point where the runtime repository is not functional on my Mac.

MarcoRossignoli · 2023-09-22T14:32:16Z

@nohwnd do you recall if we did some update to the default policy? I'd say not but maybe I'm missing something.

nohwnd · 2023-09-22T14:41:08Z

I don't think so, unless this is another bug caused by migration to arcade, but I don't see how that would happen.

vcsjones · 2023-09-22T21:11:54Z

I downgraded my local Xcode to 14 down from 15 and everything started working, so I don't think the issue is with the test host or test runner. It's something with how dotnet/runtime is getting built.

filipnavara · 2023-09-22T22:00:18Z

The new Xcode 15 linker is going to be fun.

I already submitted a fix to make NativeAOT unwinding work. Turns out that the new linker mishandles object files without the MH_SUBSECTIONS_VIA_SYMBOLS flag (.subsections_via_symbols assembly code). It ends up treating the whole object file as one function and produces unwind info only for the first function in the file.

I don't think the C compiler can produce files without the flag but I didn't check what the assembly files do. I assumed that would not necessarily matter but perhaps it was a wrong assumption for CoreCLR.

filipnavara · 2023-09-23T07:51:23Z

Can you try if filipnavara@aba0b02 helps? The CBOR tests still fail for me locally but in a different way (all the errors are related to DateTimeOffset).

vcsjones · 2023-09-23T12:58:03Z

I should have known better and thought "I bet @filipnavara knows what is going on and has probably started working on a fix already".

When I build your branch with the Xcode 15 toolchain, I get a green run of tests. The DateTimeOffset looks separate to me. I can look at that later. I have a hunch the tests fail if your timezone offset is positive.

filipnavara · 2023-09-23T12:59:06Z

Thanks for external validation!

vcsjones · 2023-09-23T23:39:32Z

Fixed by #92520.

dotnet-issue-labeler bot added the area-System.Formats.Cbor label Sep 19, 2023

ghost added the untriaged New issue has not been triaged by the area owner label Sep 19, 2023

vcsjones added os-mac-os-x macOS aka OSX area-Infrastructure and removed area-System.Formats.Cbor labels Sep 19, 2023

jkoritzinsky added area-ExceptionHandling-coreclr and removed area-Infrastructure labels Sep 19, 2023

This was referenced Sep 23, 2023

[macOS] NativeAOT unable to produce executables when using the Xcode 15 toolchain #92491

Closed

Make CoreCLR/NativeAOT assembly compile with .subsections_via_symbols on Apple platforms #92520

Merged

vcsjones changed the title ~~First-chance exceptions crashing test runner~~ Exception unwinding broken with Xcode 15 linker Sep 23, 2023

vcsjones closed this as completed Sep 23, 2023

ghost removed the untriaged New issue has not been triaged by the area owner label Sep 23, 2023

jkotas mentioned this issue Sep 24, 2023

[release/8.0] Make CoreCLR/NativeAOT assembly compile with .subsections_via_symbols on Apple platforms #92544

Merged

nil4 mentioned this issue Sep 25, 2023

Remove C++ NativeAOT workaround once it's not needed anymore royalapplications/beyondnet#60

Closed

ghost locked as resolved and limited conversation to collaborators Oct 24, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Exception unwinding broken with Xcode 15 linker #92297

Exception unwinding broken with Xcode 15 linker #92297

vcsjones commented Sep 19, 2023

ghost commented Sep 19, 2023

ghost commented Sep 19, 2023

adamsitnik commented Sep 20, 2023

vitek-karas commented Sep 20, 2023

vcsjones commented Sep 20, 2023

MarcoRossignoli commented Sep 22, 2023

nohwnd commented Sep 22, 2023

vcsjones commented Sep 22, 2023

filipnavara commented Sep 22, 2023

filipnavara commented Sep 23, 2023

vcsjones commented Sep 23, 2023 •

edited

Loading

filipnavara commented Sep 23, 2023

vcsjones commented Sep 23, 2023

Exception unwinding broken with Xcode 15 linker #92297

Exception unwinding broken with Xcode 15 linker #92297

Comments

vcsjones commented Sep 19, 2023

ghost commented Sep 19, 2023

ghost commented Sep 19, 2023

adamsitnik commented Sep 20, 2023

vitek-karas commented Sep 20, 2023

vcsjones commented Sep 20, 2023

MarcoRossignoli commented Sep 22, 2023

nohwnd commented Sep 22, 2023

vcsjones commented Sep 22, 2023

filipnavara commented Sep 22, 2023

filipnavara commented Sep 23, 2023

vcsjones commented Sep 23, 2023 • edited Loading

filipnavara commented Sep 23, 2023

vcsjones commented Sep 23, 2023

vcsjones commented Sep 23, 2023 •

edited

Loading