Windows Server 2022 - Setting affinity in .NET 6 application errors

[Marcus-Smallman]

Hi,

I have a Windows Server 2022 machine with 40 cores and 80 logical processors. Whenever I try and run my application on this machine and attempt to set it's affinity in code or via Task Manager, I get errors.

The error I get in code is:

Win32Exception (87): The parameter is incorrect

When I try and set it via Task Manager (as an admin with UAC disabled) I get:

Unable to access or set process affinity

Also, when I try and get the affinity mask for the process in code, I get 0 back.
After reading the docs:
https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-getprocessaffinitymask
https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-setprocessaffinitymask
https://docs.microsoft.com/en-us/windows/win32/procthread/processor-groups
It is clear that on Windows 11 and Server 2022 that threads are not bound to a single processor group any more if there are more than 64 processors and can run across multiple processor groups.

My theory is that I am unable to set the affinity for my application because it is performing actions on threads across multiple processor groups.
I have tried setting the affinity mask at startup in my application using:

Process proc = Process.GetCurrentProcess();
long affinityMask = (long)proc.ProcessorAffinity;
affinityMask &= 15; // First 4 processors
proc.ProcessorAffinity = (IntPtr)affinityMask;

But I just get the Win32Exception (87): The parameter is incorrect
And when I try to get the affinity mask at start up it is set to 0 meaning it was unable to get it because there are threads in multiple groups.

So for me, is it possible to set a .NET 6 application to ONLY run in a single processor group on Windows Server 2022 before the application runs, so I can then set its affinity mask?

[ghost]

Tagging subscribers to this area: @dotnet/area-system-diagnostics-process
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Hi,

I have a Windows Server 2022 machine with 40 cores and 80 logical processors. Whenever I try and run my application on this machine and attempt to set it's affinity in code or via Task Manager, I get errors.

The error I get in code is:

Win32Exception (87): The parameter is incorrect

When I try and set it via Task Manager (as an admin with UAC disabled) I get:

Unable to access or set process affinity

Also, when I try and get the affinity mask for the process in code, I get 0 back.
After reading the docs:
https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-getprocessaffinitymask
https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-setprocessaffinitymask
https://docs.microsoft.com/en-us/windows/win32/procthread/processor-groups
It is clear that on Windows 11 and Server 2022 that threads are not bound to a single processor group any more if there are more than 64 processors and can run across multiple processor groups.

My theory is that I am unable to set the affinity for my application because it is performing actions on threads across multiple processor groups.
I have tried setting the affinity mask at startup in my application using:

Process proc = Process.GetCurrentProcess();
long affinityMask = (long)proc.ProcessorAffinity;
affinityMask &= 15; // First 4 processors
proc.ProcessorAffinity = (IntPtr)affinityMask;

But I just get the Win32Exception (87): The parameter is incorrect
And when I try to get the affinity mask at start up it is set to 0 meaning it was unable to get it because there are threads in multiple groups.

So for me, is it possible to set a .NET 6 application to ONLY run in a single processor group on Windows Server 2022 before the application runs, so I can then set its affinity mask?

Author:	Marcus-Smallman
Assignees:	-
Labels:	area-System.Diagnostics.Process
Milestone:	-

[jkotas]

Related conversation: #68639 (comment)

cc @AntonLapounov @stephentoub @kouvel @Maoni0 @kunalspathak

[AntonLapounov]

Do you have DOTNET_GCCpuGroup / DOTNET_Thread_UseAllCpuGroups environment variables or corresponding configuration settings defined for your app? Setting ProcessorAffinity should work unless you have explicitly asked the runtime to use multiple processor groups via configuration settings.

[jkotas]

Setting ProcessorAffinity should work unless you have explicitly asked the runtime to use multiple processor groups via configuration settings.

Chances are that there are other non-.NET components loaded into the process that do not follow this restriction. For example, what is the Windows threadpool behavior - does it restrict itself from creating threads in multiple processor groups?

[AntonLapounov]

That OS behavior change has been implemented in backward compatible manner. A process must explicitly declare itself multigroup-aware (by calling a relevant API) to break the existing APIs like SetProcessAffinityMask. Otherwise, the legacy affinity APIs should continue to work even if the process is currently running on multiple processor groups (but not explicitly multigroup-aware).

So for me, is it possible to set a .NET 6 application to ONLY run in a single processor group

One can use start /affinity ... for that purpose.

[AntonLapounov]

or is the threadpool going to schedule them across all processor groups by default

It will use other processor groups as well; however, that doesn't make the process multigroup-aware.

FWIW, I just tried the code snippet below on an 80 core Windows Server 2022 Build 25058.1000 with the latest ARM64 SDK 6.0.302 and it worked fine:

Process proc = Process.GetCurrentProcess();
long affinityMask = (long)proc.ProcessorAffinity;
affinityMask &= 15; // First 4 processors
proc.ProcessorAffinity = (IntPtr)affinityMask;

[jkotas]

Do you have DOTNET_GCCpuGroup / DOTNET_Thread_UseAllCpuGroups environment variables or corresponding configuration settings defined for your app?

Another possibility is that something in your app calls SetThreadGroupAffinity Windows API that makes the process multigroup-aware but also breaks the old process affinity APIs. You should be able to find out by running your app under debuggers and setting a breakpoint at kernelbase!SetThreadGroupAffinity.

We would appreciate a lot if you can help us to find out what's causing the affinity setting to fail in your app.

[Marcus-Smallman]

Thanks for the responses,
The application does not have the DOTNET_GCCpuGroup / DOTNET_Thread_UseAllCpuGroups environment variables set, or the corresponding config settings.

I am also unable to run the app under a debugger on that machine as it is a machine that can not be used for development.

Another point that I should mention, is that when we downgrade the app to .NET 5, it works fine and affinity can be set.
I understand that it should work with .NET 6 and I have tried to reproduce the issue in a barebones application too, but I am unable to reproduce what we see in this specific application.

I will keep looking but if there is anything you know I could try without using a debugger, that would be really great, Thanks.

[Marcus-Smallman]

So I've been able to find the line of code in the application where by affinity cannot be set after that line has been executed.
I've then been able to reproduce the issue in a very minimal .NET 6 application, the code for you to try and reproduce follows:

namespace ProcessorAffinity
{
    using Microsoft.AspNetCore;
    using Microsoft.AspNetCore.Hosting;
    using System;

    internal class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine("Affinity can be set on the process at this point. You can try this in Task Manager. Press enter to build a web host and break setting affinity on a machine with more than 64 processors");
            var read = Console.ReadLine();


            var host = WebHost.CreateDefaultBuilder(args)
                .UseUrls($"http://*:8001")
                .UseKestrel()
                .ConfigureServices(services => { })
                .Configure((ctx, app) => { })
                .Build();

            Console.WriteLine("Affinity now CANNOT be set on this process. Try doing this in Task Manager or modifying this code to make the OS call. Press enter to exit....");

            read = Console.ReadLine();
        }
    }
}

So it appears that something in ASPNET is not multi-group aware and is breaking our application on machines with more than 64 processors.

[jkotas]

The initialization of the Quic protocol handler is calling SetThreadGroupAffinity here:

KERNELBASE!SetThreadGroupAffinity [minkernel\kernelbase\thread.c @ 3716] 
msquic!CxPlatThreadCreate+0xa2 [D:\a\_work\1\s\src\msquic\src\inc\quic_platform_winuser.h @ 889] 
msquic!CxPlatWorkersInit+0xc7 [D:\a\_work\1\s\src\msquic\src\platform\platform_worker.c @ 134] 
msquic!CxPlatInitialize+0x9c [D:\a\_work\1\s\src\msquic\src\platform\platform_winuser.c @ 358] 
msquic!MsQuicLibraryInitialize+0x4b [D:\a\_work\1\s\src\msquic\src\core\library.c @ 251] 
msquic!MsQuicAddRef+0x3a [D:\a\_work\1\s\src\msquic\src\core\library.c @ 587] 
msquic!MsQuicOpenVersion+0x54 [D:\a\_work\1\s\src\msquic\src\core\library.c @ 1546] 
System_Net_Quic!ILStubClass.IL_STUB_PInvoke(UInt32, Microsoft.Quic.QUIC_API_TABLE**)+0x6b
System_Net_Quic!System.Net.Quic.Implementations.MsQuic.Internal.MsQuicApi..cctor+0x404 [/_/src/libraries/System.Net.Quic/src/System/Net/Quic/Implementations/MsQuic/Internal/MsQuicApi.cs @ 89] 
coreclr!CallDescrWorkerInternal+0x83
coreclr!DispatchCallDebuggerWrapper+0x1c [D:\a\_work\1\s\src\coreclr\vm\callhelpers.cpp @ 156] 
coreclr!DispatchCallSimple+0x60 [D:\a\_work\1\s\src\coreclr\vm\callhelpers.cpp @ 221] 
coreclr!MethodTable::RunClassInitEx+0x164 [D:\a\_work\1\s\src\coreclr\vm\methodtable.cpp @ 3475] 
coreclr!MethodTable::DoRunClassInitThrowing+0x6df [D:\a\_work\1\s\src\coreclr\vm\methodtable.cpp @ 3658] 
coreclr!MethodTable::CheckRunClassInitThrowing+0x1b9 [D:\a\_work\1\s\src\coreclr\vm\methodtable.cpp @ 3796] 
coreclr!DynamicHelperFixup+0x7dd [D:\a\_work\1\s\src\coreclr\vm\prestub.cpp @ 3020] 
coreclr!DynamicHelperWorker+0x168 [D:\a\_work\1\s\src\coreclr\vm\prestub.cpp @ 3329] 
coreclr!DelayLoad_Helper+0x7a
System_Net_Quic!System.Net.Quic.QuicListener.get_IsSupported+0xa [/_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicListener.cs @ 15] 
Microsoft_AspNetCore_Server_Kestrel_Transport_Quic!Microsoft.AspNetCore.Hosting.WebHostBuilderQuicExtensions.UseQuic+0x11 [/_/src/Servers/Kestrel/Transport.Quic/src/WebHostBuilderQuicExtensions.cs @ 23] 

So it appears that something in ASPNET is not multi-group aware

Nit: It is the exact opposite. This code is multi-group aware and that breaks affinity setting APIs. In other words, the existing Process affinity settings API do not work on multi-group aware processes.

[Marcus-Smallman]

That makes sense. So how would one go about setting a multi-group aware process to be affinitised to the first 4 cores in a single processor group?

[jkotas]

I do not think that there is a way to do that today. The affinity has to be set during process startup or before the process becomes multi-group aware. @AntonLapounov Is that right?

[Marcus-Smallman]

Ok, thanks very much for the help. I will do some more testing to see if it can be worked around. It would maybe be a good idea to have a kestrel option to disable the use of Quic? so that when upgrading to .NET 6 it can be ignored until an appropriate time to implement support for multi-group awareness. Also, from my reading it wasn't made clear in any breaking changes document that on Windows 11/Server 2022 with more than 64 processors that upgrading to .NET 6 would break setting the process affinity after a kestrel host has been instantiated. Possibly an addition in the docs there too.

[jkotas]

It would maybe be a good idea to have a kestrel option to disable the use of Quic?

Quic is just a tip of the iceberg. There are number of other components that call SetThreadGroupAffinity API. If the app happens to use any of these components, it will again become multi-group aware. It means that setting the proces affinity using the old APIs is unlikely going to work on any large app on machines with more than 64 process.

I agree that this needs to be documented better.

[Marcus-Smallman]

I do not think that there is a way to do that today. The affinity has to be set during process startup or before the process becomes multi-group aware. @AntonLapounov Is that right?

If this is true, then that would good to know.

[AntonLapounov]

That is correct, once the process becomes multigroup-aware, the SetProcessAffinityMask Win32 function (that is what Process.ProcessorAffinity's setter uses underneath) would always fail with ERROR_INVALID_PARAMETER (error 87). Even if you successfully set an affinity mask, some components may break away by replacing that mask or calling SetThreadGroupAffinity. In case you need to enforce a limit on CPU usage, a Win32 job object or a Docker container should be used.

[AntonLapounov]

Closing this issue as external to the runtime.