MonoVM's arm64 binaries for macOS don't have Apple Silicon support

[rolfbjarne]

Description

libcorelib.dylib for microsoft.netcore.app.runtime.mono.osx-arm64 wasn't built with HAVE_PTHREAD_JIT_WRITE_PROTECT_NP set, which is required for Apple Silicon:

$ otool -p _mono_codeman_enable_write -vVt packages/microsoft.netcore.app.runtime.mono.osx-arm64/6.0.0-preview.5.21218.1/runtimes/osx-arm64/native/libcoreclr.dylib| head -6
packages/microsoft.netcore.app.runtime.mono.osx-arm64/6.0.0-preview.5.21218.1/runtimes/osx-arm64/native/libcoreclr.dylib:
(__TEXT,__text) section
_mono_codeman_enable_write:
00000000000e18a4	ret
_mono_codeman_disable_write:
00000000000e18a8	ret

runtime/src/mono/mono/utils/mono-codeman.c

Lines 661 to 675 in 9c54ae4

	void
	mono_codeman_enable_write (void)
	{
	#ifdef HAVE_PTHREAD_JIT_WRITE_PROTECT_NP
	if (__builtin_available (macOS 11, *)) {
	int level = GPOINTER_TO_INT (mono_native_tls_get_value (write_level_tls_id));
	level ++;
	mono_native_tls_set_value (write_level_tls_id, GINT_TO_POINTER (level));
	pthread_jit_write_protect_np (0);
	}
	#elif defined(HOST_MACCAT) && defined(__aarch64__)
	/* JITing in Catalyst apps is not allowed on Apple Silicon. */
	g_assert_not_reached ();
	#endif
	}

Compare with libmonosgen-2.0.dylib:

$ otool -p _mono_codeman_enable_write -vVt builds/downloads/mac-release-Darwin-b4a385816ed4f1398d0184c38f19f560e868fd80/mac-libs/libmonosgen-2.0.dylib| head -6 
../../builds/downloads/mac-release-Darwin-b4a385816ed4f1398d0184c38f19f560e868fd80/mac-libs/libmonosgen-2.0.dylib:
(__TEXT,__text) section
_mono_codeman_enable_write:
000000000028421c	stp	x20, x19, [sp, #-0x20]!
0000000000284220	stp	x29, x30, [sp, #0x10]
0000000000284224	add	x29, sp, #0x10
[...]

[ghost]

Tagging subscribers to this area: @CoffeeFlux
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

libcorelib.dylib for microsoft.netcore.app.runtime.mono.osx-arm64 wasn't built with HAVE_PTHREAD_JIT_WRITE_PROTECT_NP set, which is required for Apple Silicon:

$ otool -p _mono_codeman_enable_write -vVt packages/microsoft.netcore.app.runtime.mono.osx-arm64/6.0.0-preview.5.21218.1/runtimes/osx-arm64/native/libcoreclr.dylib| head -6
packages/microsoft.netcore.app.runtime.mono.osx-arm64/6.0.0-preview.5.21218.1/runtimes/osx-arm64/native/libcoreclr.dylib:
(__TEXT,__text) section
_mono_codeman_enable_write:
00000000000e18a4	ret
_mono_codeman_disable_write:
00000000000e18a8	ret

runtime/src/mono/mono/utils/mono-codeman.c

Lines 661 to 675 in 9c54ae4

	void
	mono_codeman_enable_write (void)
	{
	#ifdef HAVE_PTHREAD_JIT_WRITE_PROTECT_NP
	if (__builtin_available (macOS 11, *)) {
	int level = GPOINTER_TO_INT (mono_native_tls_get_value (write_level_tls_id));
	level ++;
	mono_native_tls_set_value (write_level_tls_id, GINT_TO_POINTER (level));
	pthread_jit_write_protect_np (0);
	}
	#elif defined(HOST_MACCAT) && defined(__aarch64__)
	/* JITing in Catalyst apps is not allowed on Apple Silicon. */
	g_assert_not_reached ();
	#endif
	}

Compare with libmonosgen-2.0.dylib:

$ otool -p _mono_codeman_enable_write -vVt builds/downloads/mac-release-Darwin-b4a385816ed4f1398d0184c38f19f560e868fd80/mac-libs/libmonosgen-2.0.dylib| head -6 
../../builds/downloads/mac-release-Darwin-b4a385816ed4f1398d0184c38f19f560e868fd80/mac-libs/libmonosgen-2.0.dylib:
(__TEXT,__text) section
_mono_codeman_enable_write:
000000000028421c	stp	x20, x19, [sp, #-0x20]!
0000000000284220	stp	x29, x30, [sp, #0x10]
0000000000284224	add	x29, sp, #0x10

Author:	rolfbjarne
Assignees:	-
Labels:	area-AssemblyLoader-mono, untriaged
Milestone:	-

[rolfbjarne]

CC @steveisok @akoeplinger

[filipnavara]

Interestingly the local build on M1 has HAVE_PTHREAD_JIT_WRITE_PROTECT_NP defined and builds just fine with the proper reference. Is it possible that Mono is cross-built on some older macOS machine with SDK that doesn't have the function defined?

[akoeplinger]

This doesn't seem to happen with latest builds anymore:

$ otool -p _mono_codeman_enable_write -vVt  /Users/alexander/Downloads/Microsoft.NETCore.App.Runtime.Mono.osx-arm64.6.0.0-preview.5.21268.2/runtimes/osx-arm64/native/libcoreclr.dylib | head -6
/Users/alexander/Downloads/Microsoft.NETCore.App.Runtime.Mono.osx-arm64.6.0.0-preview.5.21268.2/runtimes/osx-arm64/native/libcoreclr.dylib:
(__TEXT,__text) section
_mono_codeman_enable_write:
00000000000d520c    stp    x20, x19, [sp, #-0x20]!
00000000000d5210    stp    x29, x30, [sp, #0x10]
00000000000d5214    add    x29, sp, #0x10

[rolfbjarne]

@akoeplinger it's asserting now instead of just doing nothing:

$ otool -p _mono_codeman_enable_write -vVt ./downloads/microsoft.netcore.app.runtime.mono.maccatalyst-arm64/6.0.0-preview.6.21272.4/runtimes/maccatalyst-arm64/native/libmonosgen-2.0.dylib | head
./downloads/microsoft.netcore.app.runtime.mono.maccatalyst-arm64/6.0.0-preview.6.21272.4/runtimes/maccatalyst-arm64/native/libmonosgen-2.0.dylib:
(__TEXT,__text) section
_mono_codeman_enable_write:
00000000000d7f80	stp	x29, x30, [sp, #-0x10]!
00000000000d7f84	mov	x29, sp
00000000000d7f88	adrp	x0, 782 ; 0x3e5000
00000000000d7f8c	add	x0, x0, #0x76 ; literal pool for: "/Users/runner/work/1/s/src/mono/mono/utils/mono-codeman.c"
00000000000d7f90	mov	w1, #0x2a1
00000000000d7f94	bl	_mono_assertion_message_unreachable

[filipnavara]

@rolfbjarne It's unsupported on Mac Catalyst on Apple Silicon because pthread_jit_write_protect_np API is marked as unsupported. That's expected and Mac Catalyst apps have to be FullAOTed.

[steveisok]

There's also the JustInterp mode in the AOT compiler task made for interpreter runs.

#53197

[steveisok]

Should we close this issue then?

[rolfbjarne]

Yeah, closing this.