Update UnmanagedCallersOnlyAttribute to align with C# function pointers

[AaronRobinsonMSFT]

Background and Motivation

The C# function pointers proposal will be consuming new types as a means to indicate future calling conventions. See discussion about calling conventions with CallConvXXX referenced in dotnet/roslyn#39865.

Original API review: #32462

Proposed API

The existing API below uses the CallingConvention enumeration. This enumeration is tailored for a 32-bit Windows world. This severely limits the growth of this API.

[AttributeUsage(AttributeTargets.Method)]
public sealed class UnmanagedCallersOnlyAttribute : Attribute
{
    public UnmanagedCallersOnlyAttribute();
    public CallingConvention CallingConvention;
    public string? EntryPoint;
}

The updated API would change the CallingConvention field to be Type[] CallConvs which is more flexible as new calling conventions are adopted.

[AttributeUsage(AttributeTargets.Method)]
public sealed class UnmanagedCallersOnlyAttribute : Attribute
{
    public UnmanagedCallersOnlyAttribute();
    public Type[]? CallConvs;
    public string? EntryPoint;
}

Usage Examples

Old:

[UnmanagedCallersOnly(CallingConvention = CallingConvention.StdCall)]
public static int SquareNumber(int val)
{
    return val * val;
}

New:

using System.Runtime.CompilerServices;

[UnmanagedCallersOnly(CallConvs = new [] { typeof(CallConvStdcall) })]
public static int SquareNumber(int val)
{
    return val * val;
}

Note It would be requested that the Roslyn compiler apply the same validation algorithm applied in the function pointers specification to the types in the CallConvs array.

Risks

If this work is not done for .NET 5 updating the UnmanagedCallersOnlyAttribute will become substantially more difficult.

[AaronRobinsonMSFT]

/cc @333fred @jkotas @elinor-fung @jkoritzinsky

[tannergooding]

No types corresponds to unmanaged (WinApi), is that correct?

We can't easily expose convenience constructors that take CallingConvention (and maybe bool for things like SupressGCTransition) because the compiler would have to understand that constructor, is that correct?

[333fred]

We can't easily expose convenience constructors that take CallingConvention (and maybe bool for things like SupressGCTransition) because the compiler would have to understand that constructor, is that correct?

That's correct. If we have to understand that enum, it defeats the purpose of having an extensible calling convention protocol in the first place :).

[AaronRobinsonMSFT]

No types corresponds to unmanaged (WinApi), is that correct?

Correct.

SupressGCTransition

I realize that this is part of the function pointers discussion, but will call out that marking the function UnmanagedCallersOnly with SupressGCTransition would be a very unusual and bad combination.

[AaronRobinsonMSFT]

@tannergooding and @333fred Thoughts on changing the CallingConventionTypes property to CallConv to match the targeted type prefixes?

[tannergooding]

Maybe CallConvs? to make it clearer that it is a collection?

[333fred]

However many 🚲s you want to 🏠 (no :shed: 🙁) is fine with me. So long as there's an array of types that matches up to what we need to emit.

[andrew-boyarshin]

I know I'm a bit too late (API is already approved), but here's my 2 cents: I'd name it CallingConventions, rather than CallConvs, even though it contains types named CallConvXXX. Abbreviations obscure meaning.

[nathan-alden-sr]

FWIW, Supress is spelled wrong. Hopefully wherever that is can be corrected.

[AaronRobinsonMSFT]

@nathan-alden-sr It is correct in the official name - https://github.com/dotnet/runtime/blob/master/src/libraries/System.Private.CoreLib/src/System/Runtime/InteropServices/SuppressGCTransitionAttribute.cs. I am just poor at spelling in comments 😄