.NET 6 RC2 Update for macOS and Windows Arm64

[richlander]

.NET 6 RC2 Update for macOS and Windows Arm64

Update: Newer support information is now available at #22380. The new information supersedes the support information here. In particular, the .NET 5 Arm64 SDK WILL BE supported for its full stated support timeframe.

We have good news. The project to support macOS and Windows arm64 is almost done. We've been working on it for over a year, with help when we needed it from the fine folks on the macOS and the Windows teams. At first, we thought that the project was solely about supporting Arm64 on macOS and that Rosetta 2 would cover x64. Easy and just about ISAs, right? Not quite. As we dug deeper, it became clear that x64 + Arm64 co-existence was the bigger task to take on, focused on the CLI and installers, including a few places where they need to agree. Anyway, we're close to delivering builds of .NET that we think deliver the best experience we could imagine.

Before I dive into the details, I'll provide a quick summary of where we are at for macOS and Windows Arm64 machines:

• .NET 6 RC2 enables Arm64 + x64 co-existence by installing Arm64 and x64 builds to different locations. Until now, Arm64 and x64 builds overwrite each other, which led to general sadness.
• You need to uninstall all .NET builds and start from scratch (on macOS and Windows Arm64 machines) to adopt .NET 6 RC2+. This includes Arm64 and x64, .NET 6 and pre-.NET 6.
• Pre .NET 6 builds are not yet ready to install.
• The CLI enables you to use the Arm64 SDK to do Arm64 AND x64 development (assuming you have the required Arm64 and x64 runtimes installed). The same is true in reverse.
• We want people to just use the Arm64 SDK since it will be a better experience (native architecture performance; one SDK to maintain). We will continue to improve the product to make this model the easy choice for most developers.
• For the SDK, we will only support .NET 6+ on Arm64. Earlier SDK builds will be blocked on Arm64.
• For runtimes, we will support all in-support versions, Arm64 and x64.
• .NET 6 RC2 delivers the bulk of the final .NET 6 experience for Arm64 (including x64 emulation).
• We hope to update .NET Core 3.1 and .NET 5 runtimes to align with .NET 6 RTM (both in time and with the technical requirements). That's still TBD.
• RC2 nightly builds are currently broken, so you will need to wait a couple more weeks until we actually ship RC2 to try this all out.
• The .NET 5 SDK for Windows Arm64 will go out of support early, with .NET 6 RTM.

We're also considering making breaking changes to dotnet test to unify the arguments we use for architecture targeting:

• [Breaking change] For dotnet test, switch -a to alias --arch instead of --test-adapter-path #21389
• [Breaking change] For dotnet test, switch -r to alias --runtime instead of --results-dir #21952

If you want more details, you can check out our x64 emulation plan. I believe that document remains accurate to our final implementation.

Installers

About twenty years ago, the Windows team started shipping x64 builds with this really great WoW64 (Windows 32 on Windows 64) technology that enabled 32-bit x86 apps to run on an x64 OS and also provided (and this is the key point) file system and registry virtualization. It was obvious from day one that 32-bit apps should install into C:\Program Files (x86) and 64-bit apps should install to C:\Program Files, because that was the formal well-defined model provided by the Windows team. Years later, Visual Studio 2019 still relies on 32-bit support in Windows using this same system. Neither macOS or Windows has a similar model for Arm64. As a result, we had to invent our own approach.

We decided that:

• The native architecture build of .NET will always install into the dotnet folder.
• It is OK for the emulated build to be a second-class citizen, and install in a less-good location.
• It is OK for the x64 installers to have to pivot between two locations: the dotnet folder on x64 machines (where x64 is the native architecture), and a new location on Arm64 machines.
• It is NOT OK to produce two x64 builds, one for x64 machines and another for Arm64 machines.
• We will only add the native architecture build to the PATH, even if you only install the emulated build.

That led us to the following model:

• Arm64 builds will install to dotnet (as already described) and add that directory to the PATH.
• x64 builds will install to dotnet/x64 on Arm64 machines and not modify the PATH.

That means if you:

• Install both Arm64 and x64 builds, dotnet will be the Arm64 one.
• Install just an Arm64 build, dotnet will be the Arm64 one.
• Install just an x64 build, dotnet won't be in the path so won't work.
• You can use the x64 dotnet via a variety of ways, some inconvenient (absolute path usage) and others pretty workable (add the x64 dotnet to the front of the PATH temporarily or symbolic links). These are classic operating system techniques that have been used for many years for similar scenarios.

You can see this model demonstrated:

rich@MacBook-Air-M1-2020 ~ % dotnet --info | grep RID
 RID:         osx-arm64
rich@MacBook-Air-M1-2020 ~ % /usr/local/share/dotnet/x64/dotnet --info | grep RID
 RID:         osx-x64

The following is the simplest way to use the x64 SDK, most appropriate for temporary use.

rich@MacBook-Air-M1-2020 ~ % export PATH=/usr/local/share/dotnet/x64:$PATH 
rich@MacBook-Air-M1-2020 ~ % dotnet --info | grep RID
 RID:         osx-x64

You can also use the following PATH flipping trick to go back-and-forth.

rich@MacBook-Air-M1-2020 ~ % dotnet --info | grep RID
 RID:         osx-arm64
rich@MacBook-Air-M1-2020 ~ % export OLDPATH=$PATH
rich@MacBook-Air-M1-2020 ~ % export PATH=/usr/local/share/dotnet/x64:$PATH
rich@MacBook-Air-M1-2020 ~ % dotnet --info | grep RID                     
 RID:         osx-x64
rich@MacBook-Air-M1-2020 ~ % export PATH=$OLDPATH                      
rich@MacBook-Air-M1-2020 ~ % dotnet --info | grep RID
 RID:         osx-arm64
rich@MacBook-Air-M1-2020 ~ % 

The following is another approach for using the x64 SDK, with symbolic links, as a more permanent pattern.

rich@MacBook-Air-M1-2020 ~ % sudo ln -s /usr/local/share/dotnet/x64/dotnet /usr/local/bin/dotnetx64
rich@MacBook-Air-M1-2020 ~ % dotnetx64 --info | grep RID
 RID:         osx-x64

There are (mostly) no shared files between the Arm64 and x64 builds. That means that you can uninstall an x64 build and not worry about it affecting the Arm64 build, and vice versa.

These changes are significant and are breaking relative to the existing model. You need to uninstall/remove all .NET builds (x64 and Arm64) on your machine in order to move to this model. There is no migration tool. You need to remove .NET from your machine and start again. We're sorry about that, however, we could not have delivered this model if we'd had to maintain compatibility. We also believe that we are still very early with Arm64 (on macOS and Windows) so felt that the break was OK.

Apps

Apps are built for one OS+arch combination. This is primarily due to the apphost that is created for each app, which is inherently OS+arch-specific. If you build an x64 app and copy it to an Arm64 machine, it will run as x64. There is no magic conversion to Arm64. For example, when running under Rosetta 2, an x64 apphost will attempt to find a compatible x64 runtime. An Arm64 runtime will not be used nor is it usable in that scenario.

Hosts

Application hosts (apphost) needs to be taught about where to look for .NET builds on the machine, particularly for x64. We could have done this several different ways. We already had significant functionality for locating .NET (beyond the PATH). We built on it.

The DOTNET_ROOT ENV is used by hosts to locate a .NET installation for private installs. That can be ambiguous. We now have multiple variants:

• DOTNET_ROOT -- Used to locate the .NET install location.
• DOTNET_ROOT_X64 -- Used to locate the x64 install location, in particular.
• DOTNET_ROOT_ARM64 -- Used to locate the Arm64 install location, in particular.

In general, you should just use DOTNET_ROOT. The arch-specific cases are only needed if you have x64 AND Arm64 private installs. The arch-specific ENVs are preferred over DOTNET_ROOT if they are set.

Note: We also updated the 32-bit/x86 ENV to match, with DOTNET_ROOT_X86. It was previously DOTNET_ROOT(x86). That's still supported but deprecated. That form clearly has heritage with Windows, and we were not going to create a DOTNET_ROOT(x64) to continue with that theme. That form is also harder to type.

As suggested, those ENVs are great for private .NET installs, but not intended for general use. For that, the installers write state in an OS-idiomatic way to inform hosts where to look for .NET builds.

This is the state for macOS.

rich@MacBook-Air-M1-2020 ~ % ls /etc/dotnet                         
install_location	install_location_arm64	install_location_x64
rich@MacBook-Air-M1-2020 ~ % cat /etc/dotnet/install_location
/usr/local/share/dotnet/x64
rich@MacBook-Air-M1-2020 ~ % cat /etc/dotnet/install_location_x64 
/usr/local/share/dotnet/x64
rich@MacBook-Air-M1-2020 ~ % cat /etc/dotnet/install_location_arm64 
/usr/local/share/dotnet

The Arm64 installers write the install_location_arm64 file, while x64 installers writes the install_location_x64 file. Both installers write the install_location file. That's the one shared asset between the two installers, which I hinted at earlier.

Existing x64 apps expect the install_location file. We need to keep it as x64-oriented for compat. Going forward, this file is deprecated. We are moving to the architecture-specific files as the new model. That removes ambiguity. We will stop writing the install_location file with either .NET 7 or 8.

CLI

As suggested, we want it be easy and pleasant to use the Arm64 SDK for both Arm64 and x64 development. It naturally follows that the CLI needs first-class architecture targeting to enable that (which it previous has not had). We had to make several changes (some of which will land in .NET 7) to enable that.

There were a few things to deliver:

• Architecture targeting is available for all relevant CLI commands.
• Architecture targeting doesn't require using RIDs.
• Specifying an architecture doesn't switch your deployment type (for example framework-dependent -> self-contained).

We invented two new flags to satisfy these requirements:

• --arch or -a is used to specify the desired architecture, like -a arm64 or -a x64. The architecture of the given SDK is the default.
• --os is used to specify the desired operating system, like --os osx or --os win. The OS of the given SDK is the default. We don't expect this flag to be used nearly as often. It provided mostly for completeness.

Close readers will realize that -a and --os are just the two different sides of a RID. Whenever you specify one, the other half is taken from the SDK you are using, resulting in a complete RID. You can also specify both, to fully replace using -r.

Today (and continuing in .NET 6), when you specify a rid (like -r osx-arm64), you produce a self-contained application by default. That's a very unfortunate default and an early design decision (that at least I regret). Part of the reason for inventing -a was to change this behavior in a non-breaking way. When you use -a, it doesn't change the deployment type of your app.

Early in .NET 7, we will make a breaking change to -r to make it match this new behavior, which we believe is much better. That change will remove the need for the --no-self-contained flag, which is partial evidence of its value.

Let's take a look at some examples. Imagine you build and test your .NET 6 app with the Arm64 SDK and runtimes, but just want to run or test the app occasionally with x64. That's straightforward with this new model.

rich@MacBook-Air-M1-2020 app % cat Program.cs 
using System.Runtime.InteropServices;

Console.WriteLine($"Hello, {RuntimeInformation.OSArchitecture}!");
rich@MacBook-Air-M1-2020 app % cat app.csproj | grep Target
    <TargetFramework>net6.0</TargetFramework>
rich@MacBook-Air-M1-2020 app % dotnet run
Hello, Arm64!
rich@MacBook-Air-M1-2020 app % dotnet run -a x64
Hello, X64!
rich@MacBook-Air-M1-2020 app % ls bin/Debug/net6.0 
app			app.pdb			ref
app.deps.json		app.runtimeconfig.json
app.dll			osx-x64
rich@MacBook-Air-M1-2020 app % ls bin/Debug/net6.0/osx-x64 
app			app.dll			app.runtimeconfig.json
app.deps.json		app.pdb			ref

We can try the same thing with the x64 SDK.

rich@MacBook-Air-M1-2020 app % export PATH=/usr/local/share/dotnet/x64:$PATH 
rich@MacBook-Air-M1-2020 app % dotnet --info | grep RID
 RID:         osx-x64
rich@MacBook-Air-M1-2020 app % dotnet run         
Hello, X64!
rich@MacBook-Air-M1-2020 app % dotnet run -a arm64
Hello, Arm64!

You can see that this same behavior is provided for other verbs, like dotnet watch:

The CLI has special treatment for earlier .NET versions since they do not support Arm64.

• If there is only one architecture available for a given .NET version, the SDK will target the matching RID by default.
• If there are multiple available, then the SDK will target its RID.

In practice, that means:

• On macOS, .NET 6+ apps default to the SDK RID, while earlier versions default to x64.
• On Windows, the model is the same except that .NET 5+ apps default to the SDK RID. We added support to .NET 5 for Windows Arm64 with the 5.0.8 patch.

The following demonstrates that behavior, with .NET 5 using the .NET 6 Arm64 SDK (on macOS):

rich@MacBook-Air-M1-2020 app % dotnet --info | grep RID
RID: osx-arm64
rich@MacBook-Air-M1-2020 app % cat app.csproj| grep Target
<TargetFramework>net5.0</TargetFramework>
rich@MacBook-Air-M1-2020 app % dotnet run
Hello, X64 from .NET 5.0.10!
rich@MacBook-Air-M1-2020 app % /usr/local/share/dotnet/x64/dotnet run
Hello, X64 from .NET 5.0.10!

And then with .NET 6:

rich@MacBook-Air-M1-2020 app % cat app.csproj| grep Target
<TargetFramework>net6.0</TargetFramework>
rich@MacBook-Air-M1-2020 app % dotnet run
Hello, Arm64 from .NET 6.0.0-rc.2.21474.18!
rich@MacBook-Air-M1-2020 app % /usr/local/share/dotnet/x64/dotnet run
Hello, X64 from .NET 6.0.0-rc.2.21474.18!

Tool installation

You can use the -a argument for dotnet tool install, just like dotnet run.

The dotnet-serve tool is a good example to play with since it currently targets .NET 5 and doesn't roll-forward.

rich@MacBook-Air-M1-2020 ~ % dotnet tool install -g dotnet-serve -a x64
Skip NuGet package signing validation. NuGet signing validation is not available on Linux or macOS https://aka.ms/workloadskippackagevalidation .
You can invoke the tool using the following command: dotnet-serve
Tool 'dotnet-serve' (version '1.9.71') was successfully installed.
rich@MacBook-Air-M1-2020 ~ % dotnet-serve 
Starting server, serving .
Listening on:
  http://localhost:50114

Press CTRL+C to exit

The problem with this model is that you have to know key details about the tool ahead-of-time in order to install it correctly. You will see the following behavior w/o the correct -a argument.

rich@MacBook-Air-M1-2020 ~ % dotnet tool install -g dotnet-serve     
You can invoke the tool using the following command: dotnet-serve
Tool 'dotnet-serve' (version '1.9.71') was successfully installed.
rich@MacBook-Air-M1-2020 ~ % dotnet-serve                       
zsh: killed     dotnet-serve
rich@MacBook-Air-M1-2020 ~ % codesign -s - .dotnet/tools/dotnet-serve  
rich@MacBook-Air-M1-2020 ~ % dotnet-serve                            
It was not possible to find any compatible framework version
The framework 'Microsoft.AspNetCore.App', version '5.0.0' (arm64) was not found.

That's a pretty bad experience. We're still working on fixing that.

.NET 5 SDK for Windows Arm64

As stated above, we will only support .NET 6+ SDKs on Windows Arm64. That means that will not support the .NET 5 Arm64 SDK. On the surface, that's strange since .NET 5 is supported until May 2022. We decided to simplify the project and make a blanket choice to support only .NET 6+ SDKs for both macOS and Windows.

That means that we'll service the .NET 5 Arm64 SDK in October 2021 for the last time and will no longer build or support it after .NET 6 is released. For folks that are using that SDK, we hope it's not too much to ask to move to the .NET 6 SDK. If it a challenge, we're sorry.

.NET 5 runtimes (x64 and Arm64) will be supported for the remainder of the .NET 5 support period. This change in support is specific to the .NET 5 SDK for Windows Arm64.

Windows 11 Arm64

This document uses macOS for all the examples. We've been most focused on macOS since it is already in market and presents additional challenges (like notarization). We are also validating this same set of experiences on Windows 11 Arm64. We don't expect any Windows 11-specific challenges.

.NET 6 RC2 installers have been updated in the same way for both macOS and Windows.

What's left?

• dotnet test hasn't been updated yet. For now, if you want to test with x64, you will need to use the x64 SDK. At present, it doesn't look like this will be resolved by .NET 6 RTM.
• dotnet tool install needs to be updated to pick a compatible apphost, in much the same way as dotnet run works.
• dotnet tool install needs to codesign the apphost (at least for Arm64), much like you've seen demonstrated in this document.

There are some trailing related bugs that will get resolved by .NET 6 RTM or later with .NET 6.0.200.

• Self-contained publish/build error #21677
• Implicit architecture targeting not in place for dotnet tool install #21660
• Enable implicit RID with --sc #21405
• [Breaking change] For dotnet test, switch -a to alias --arch instead of --test-adapter-path #21389

/cc @sfoslund, @ericstj, @vitek-karas

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[Mercurial]

I haven't been following dotnet so much in terms of platform compatibility but this is a great information about it of what being developed and planned.

My question is that how is linux x64 and arm going to look like? 🙋

Thanks!

[richlander]

My question is that how is linux x64 and arm going to look like?

That's a great question. We've had Linux Arm64 since .NET Core 3.0 (and x64 since 1.0). The emulation experience on Linux is totally different and doesn't have the challenges that macOS and Windows present. As a result, we have no similar plans for Linux.

The following bug is the primary issue (that I'm aware of) for Linux processor emulation for .NET: https://gitlab.com/qemu-project/qemu/-/issues/249. If that's an issue for you/anyone, please show your support for it.

[riverar]

As stated above, we will only support .NET 6+ SDKs on Windows Arm64. That means that will not support the .NET 5 Arm64 SDK. On the surface, that's strange since .NET 5 is supported until May 2022. We decided to simplify the project and make a blanket choice to support only .NET 6+ SDKs for both macOS and Windows.

Hi @richlander, is there any more information on the reasoning behind breaking support promises for the ARM64 SDK? The provided reason--"we decided to simplify the project"--is pretty flimsy.

[richlander]

This is a big project, as this post demonstrates. I wanted to find ways to scope it to make it cheaper and simpler to understand. I believe this plan does that.

The .NET 5 Arm64 SDK was only released in July. My (unproven) belief is that few folks have a hard dependency on it. If folks have a hard dependency on it and cannot reasonably move to .NET 6, I would like to know. This is my MO .... cut the project back to its most limited form that fits the bill, share the plan, and see what people have to say.

It's unfortunate to break support contracts. We don't normally do that. I think this is one of those times where our efforts are better spent elsewhere. If folks raise their hand and describe why moving to .NET 6 is unreasonable, we'll reconsider. If no one raises their hand, we'll go forward with the currently plan.

Stepping back, if folks need the .NET 5 Arm64, we'll only support it on Windows 10. We're not going to support it on Windows 11. I don't see a reason to do that.

[riverar]

This was brought to my attention, funny enough, not because of the lack of SDK support but rather community concerns around the durability of future .NET lifecycle promises. I appreciate the pragmatic approach here though and agree this is probably a non-event.

[richlander]

Totally get it. If we do this more, I would be more concerned. I doubt you'll see that.

FWIW ... our leadership wasn't very keen on this idea. This isn't their preferred approach.

[sschultze]

I am unable to get this to work on my Mac (MacBook Air M1, macOS 11.6).

What I did:

• Removed all SDKs: sudo ./dotnet-core-uninstall remove --sdk --all
• Removed all runtimes: sudo ./dotnet-core-uninstall remove --runtime --all
• Cleaned up the rest: sudo rm -rf /usr/local/share/dotnet/
• Downloaded and ran the .NET 6 RC 2 SDK installer for macOS - ARM64
• Extracted the ASP.NET Core Runtime 6.0.0-rc.2 - X64 into /usr/local/share/dotnet/x64/sdk (unfortunately there's no installer)
• Ensured that "Allow apps downloaded from" is set to "App Store and identified developers"
• Un-quarantined all files as a possible way to solve my problem: xattr -r -d /usr/local/share/dotnet/x64
• Tried to run my project using the --arch flag: dotnet run --arch x64 --project ./src/Server/Server.csproj -c Debug

The error I am getting is:

Failed to load /usr/local/share/dotnet/x64/shared/Microsoft.NETCore.App/6.0.0-rc.2.21480.5/libhostpolicy.dylib, error: dlopen(/usr/local/share/dotnet/x64/shared/Microsoft.NETCore.App/6.0.0-rc.2.21480.5/libhostpolicy.dylib, 1): no suitable image found. Did find:
/usr/local/share/dotnet/x64/shared/Microsoft.NETCore.App/6.0.0-rc.2.21480.5/libhostpolicy.dylib: code signature in (/usr/local/share/dotnet/x64/shared/Microsoft.NETCore.App/6.0.0-rc.2.21480.5/libhostpolicy.dylib) not valid for use in process using Library Validation: library load disallowed by system policy
An error occurred while loading required library libhostpolicy.dylib from [/usr/local/share/dotnet/x64/shared/Microsoft.NETCore.App/6.0.0-rc.2.21480.5]

Am I doing anything wrong or is there a bug?

[vitek-karas]

@ericstj for the installers

Maybe you could try installing the x64 6.0 RC2 SDK installer (which I think includes the ASP.NET runtime)?

[sschultze]

@vitek-karas @richlander says "We want people to just use the Arm64 SDK since it will be a better experience." and "The CLI enables you to use the Arm64 SDK to do Arm64 AND x64 development (assuming you have the required Arm64 and x64 runtimes installed)."

So, using the Arm64 SDK is the thing to do.

Or do you mean that I should install the x64 SDK in addition (instead of unzipping the x64 runtime)? I don't have an M1 Mac to test this today, but if I recall correctly from the weekend, I tried this and it replaced the Arm64 SDK, which is not what I wanted. (But I am not 100% sure on this.)

[vitek-karas]

I only meant it as a workaround for the problematic ASP.NET installer - We do still recommend to use the arm64 SDK for all your work (it should work targeting x64).

[sschultze]

@vitek-karas Thank you, but my (currently absolutely satisfiable) workaround is to just stay on RC 1 until this is fixed. :-)

[sschultze]

I have created a separate issue for this:

#22176

[lucas-zimerman]

I had some unneeded trouble setting up .NET Core 2.X, 3.X nd 5.X to work together with .NET 6 on Arm.

I was able to solve it the following way:

• nuke/delete usr/local/share/dotnet
• Install the old .NET Core SDKs (on my case I installed in the following order: 2.1.202, 2.1.818, 3.1.414, 5.0.402)
• Done that, make a copy of the folder usr/local/share/dotnet and delete the folder from usr/local/share/dotnet
• Install .NET 6.0 SDK for Arm.
• goto usr/local/share/dotnet, create a folder called x64 and paste everything that was previously copied.
  with that you end up with a folder like the following:
  ...\x64\templates, '...\x64\sdk' ,...
• Now proceed to install the .NET 6.0 SDK for x64.
• Not sure why but doing the previous install removes dotnet from the arm application, so reinstall the .NET 6.0 RC2 SDK for Arm.

Doing that, I was able to test my code on legacy Frameworks but also the newer ones like .NET 6.

Keep in mind that dotnet test is broken so you'll need to run 'usr/local/share/dotnet/x64/dotnet test ...' in order to test the older Frameworks.

[richlander]

Please use #22380 for further discussion on this topic.

[mattjohnsonpint]

@lucas-zimerman - I found a better approach, which does basically the same thing. See #22380 (comment)