[API Proposal]: Implementations of marshallers using CustomTypeMarshallerAttribute

[elinor-fung]

Background and motivation

For source-generated p/invokes, #46838 and #66121 were recently approved to support custom marshalling via types marked with System.Runtime.InteropServices.CustomTypeMarshallerAttribute and conforming to a specific shape. This issue proposes adding implementations of marshallers that the p/invoke source generator would be able use.

API Proposal

Strings:

[CLSCompliant(false)]
[CustomTypeMarshaller(typeof(string), BufferSize = 0x100,
    Features = CustomTypeMarshallerFeatures.UnmanagedResources | CustomTypeMarshallerFeatures.CallerAllocatedBuffer | CustomTypeMarshallerFeatures.TwoStageMarshalling )]
public unsafe ref struct AnsiStringMarshaller
{
    public AnsiStringMarshaller(string? str);
    public AnsiStringMarshaller(string? str, Span<byte> buffer);

    public byte* ToNativeValue();
    public void FromNativeValue(byte* value);

    public string? ToManaged();

    public void FreeNative();
}

[CLSCompliant(false)]
[CustomTypeMarshaller(typeof(string), BufferSize = 0x100,
    Features = CustomTypeMarshallerFeatures.UnmanagedResources | CustomTypeMarshallerFeatures.CallerAllocatedBuffer | CustomTypeMarshallerFeatures.TwoStageMarshalling )]
public unsafe ref struct Utf8StringMarshaller
{
    public Utf8StringMarshaller(string? str);
    public Utf8StringMarshaller(string? str, Span<byte> buffer);

    public byte* ToNativeValue();
    public void FromNativeValue(byte* value);

    public string? ToManaged();

    public void FreeNative();
}

[CLSCompliant(false)]
[CustomTypeMarshaller(typeof(string), BufferSize = 0x100,
    Features = CustomTypeMarshallerFeatures.UnmanagedResources | CustomTypeMarshallerFeatures.CallerAllocatedBuffer | CustomTypeMarshallerFeatures.TwoStageMarshalling)]
public unsafe ref struct Utf16StringMarshaller
{
    public Utf16StringMarshaller(string? str);
    public Utf16StringMarshaller(string? str, Span<ushort> buffer);

    public ref ushort GetPinnableReference();

    public ushort* ToNativeValue();
    public void FromNativeValue(ushort* value);

    public string? ToManaged();

    public void FreeNative();
}

Arrays:

[CustomTypeMarshaller(typeof(CustomTypeMarshallerAttribute.GenericPlaceholder[]),
    CustomTypeMarshallerKind.LinearCollection, BufferSize = 0x200,
    Features = CustomTypeMarshallerFeatures.UnmanagedResources | CustomTypeMarshallerFeatures.CallerAllocatedBuffer | CustomTypeMarshallerFeatures.TwoStageMarshalling)]
public unsafe ref struct ArrayMarshaller<T>
{
    public ArrayMarshaller(int sizeOfNativeElement);
    public ArrayMarshaller(T[]? array, int sizeOfNativeElement);
    public ArrayMarshaller(T[]? array, Span<byte> buffer, int sizeOfNativeElement);

    public ReadOnlySpan<T> GetManagedValuesSource();

    public Span<T> GetManagedValuesDestination(int length);
    public Span<byte> GetNativeValuesDestination();

    public ReadOnlySpan<byte> GetNativeValuesSource(int length);

    public ref byte GetPinnableReference();

    public byte* ToNativeValue();
    public void FromNativeValue(byte* value);

    public T[]? ToManaged();

    public void FreeNative();
}

[CustomTypeMarshaller(typeof(CustomTypeMarshallerAttribute.GenericPlaceholder*[]),
    CustomTypeMarshallerKind.LinearCollection, BufferSize = 0x200,
    Features = CustomTypeMarshallerFeatures.UnmanagedResources | CustomTypeMarshallerFeatures.CallerAllocatedBuffer | CustomTypeMarshallerFeatures.TwoStageMarshalling)]
public unsafe ref struct PtrArrayMarshaller<T> where T : unmanaged
{
    public PtrArrayMarshaller(int sizeOfNativeElement);
    public PtrArrayMarshaller(T*[]? array, int sizeOfNativeElement);
    public PtrArrayMarshaller(T*[]? array, Span<byte> buffer, int sizeOfNativeElement);

    public ReadOnlySpan<IntPtr> GetManagedValuesSource();

    public Span<IntPtr> GetManagedValuesDestination(int length);
    public Span<byte> GetNativeValuesDestination();

    public ReadOnlySpan<byte> GetNativeValuesSource(int length);
    
    public ref byte GetPinnableReference();

    public byte* ToNativeValue();
    public void FromNativeValue(byte* value);

    public T*[]? ToManaged();

    public void FreeNative();
}

API Usage

The p/invoke source generator would use these marshallers for marshalling some types without requiring explicit user specification of the custom marshaller. The marshallers could also be explicitly used with the LibraryImportAttribute.StringMarshallingCustomType property or the MarshalUsing and NativeMarshalling attributes.

[LibraryImport("NativeLib")]
internal static partial void Method([MarshalUsing(typeof(AnsiStringMarshaller))] string str);

[LibraryImport("NativeLib", StringMarshallingCustomType = typeof(AnsiStringMarshaller))]
internal static partial string Method(string s1, string s2);

Alternative Designs

The p/invoke source generator could inject marshallers / emit the code directly instead of using public marshaller types. However, that would be more of a size impact than having the types in runtime libraries. It would also mean that any issues in sensitive areas like string and array marshalling would require users to recompile to regenerate code, rather than being fixed by a newer runtime.

Risks

No response

[ghost]

Tagging subscribers to this area: @dotnet/interop-contrib
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Background and motivation

For source-generated p/invokes, #46838 and #66121 were recently approved to support custom marshalling via types marked with System.Runtime.InteropServices.CustomTypeMarshallerAttribute. This issue proposes adding implementations of marshallers that the p/invoke source generator would be able use.

API Proposal

Strings:

[CustomTypeMarshaller(typeof(string), BufferSize = 0x100,
    Features = CustomTypeMarshallerFeatures.UnmanagedResources | CustomTypeMarshallerFeatures.CallerAllocatedBuffer | CustomTypeMarshallerFeatures.TwoStageMarshalling )]
public unsafe ref struct AnsiStringMarshaller
{
    public AnsiStringMarshaller(string? s);
    public AnsiStringMarshaller(string? s, Span<byte> buffer);

    public byte* ToNativeValue();
    public void FromNativeValue(byte* value);

    public string? ToManaged();

    public void FreeNative();
}

[CustomTypeMarshaller(typeof(string), BufferSize = 0x100,
    Features = CustomTypeMarshallerFeatures.UnmanagedResources | CustomTypeMarshallerFeatures.CallerAllocatedBuffer | CustomTypeMarshallerFeatures.TwoStageMarshalling )]
public unsafe ref struct Utf8StringMarshaller
{
    public Utf8StringMarshaller(string? s);
    public Utf8StringMarshaller(string? s, Span<byte> buffer);

    public byte* ToNativeValue();
    public void FromNativeValue(byte* value);

    public string? ToManaged();

    public void FreeNative();
}

[CustomTypeMarshaller(typeof(string), BufferSize = 0x200,
    Features = CustomTypeMarshallerFeatures.UnmanagedResources | CustomTypeMarshallerFeatures.CallerAllocatedBuffer | CustomTypeMarshallerFeatures.TwoStageMarshalling)]
public unsafe ref struct Utf16StringMarshaller
{
    public Utf16StringMarshaller(string? s);
    public Utf16StringMarshaller(string? s, Span<byte> buffer);

    public ref ushort GetPinnableReference();

    public ushort* ToNativeValue();
    public void FromNativeValue(ushort* value);

    public string? ToManaged();

    public void FreeNative();
}

Arrays:

[CustomTypeMarshaller(typeof(CustomTypeMarshallerAttribute.GenericPlaceholder[]), CustomTypeMarshallerKind.LinearCollection, BufferSize = 0x200,
    Features = CustomTypeMarshallerFeatures.UnmanagedResources | CustomTypeMarshallerFeatures.CallerAllocatedBuffer | CustomTypeMarshallerFeatures.TwoStageMarshalling)]
public unsafe ref struct ArrayMarshaller<T>
{
    public ArrayMarshaller(int sizeOfNativeElement);
    public ArrayMarshaller(T[]? managed, int sizeOfNativeElement);
    public ArrayMarshaller(T[]? managed, Span<byte> stackSpace, int sizeOfNativeElement);

    public ReadOnlySpan<T> GetManagedValuesSource();

    public Span<T> GetManagedValuesDestination(int length);
    public Span<byte> GetNativeValuesDestination();

    public ReadOnlySpan<byte> GetNativeValuesSource(int length);

    public ref byte GetPinnableReference();

    public byte* ToNativeValue();
    public void FromNativeValue(byte* value);

    public T[]? ToManaged();

    public void FreeNative();
}

API Usage

The p/invoke source generator would use these marshallers for marshalling some types without requiring explicit user specification of the custom marshaller. The marshallers could also be explicitly used with the LibraryImportAttribute.StringMarshallingCustomType property or the MarshalUsing and NativeMarshalling attributes.

[LibraryImport("NativeLib")]
internal static partial void Method([MarshalUsing(typeof(AnsiStringMarshaller))] string str);

[LibraryImport("NativeLib", StringMarshallingCustomType = typeof(AnsiStringMarshaller))]
internal static partial string Method(string s1, string s2);

Alternative Designs

The p/invoke source generator could inject marshallers / emit the code directly instead of using public marshaller types. However, that would be more of a size impact than having the types in runtime libraries. It would also mean that any issues in sensitive areas like string and array marshalling would require users to recompile to regenerate code, rather than being fixed by a newer runtime.

Risks

No response

Author:	elinor-fung
Assignees:	-
Labels:	api-suggestion, area-System.Runtime.InteropServices
Milestone:	-

[jkoritzinsky]

We should also include the PtrArrayMarshaller<T> type in the proposal so we can handle all array types that users may define.

[elinor-fung]

Updated to include PtrArrayMarshaller<T>.

[Sergio0694]

Long shot, but could we use this as another argument to get more support for #13627? Adding a whole separate type just to support pointers feels like a hack (I mean maybe not a hack, but it is a workaround and sure looks like one), and it also still wouldn't really support all possible scenarios given that it'd only restrict that to arrays of single pointers, nor arrays of pointers to void 🤷‍♂️

[jkoritzinsky]

I mentioned it to @jaredpar a while back the last time it came up on the C# discord. Don't remember what his response though.

[jkotas]

Utf16StringMarshaller Span buffer

Should the buffer be strongly typed (ie Span<ushort>) to communicate the required alignment? It probably does not matter a whole lot for this specific case, but it may matter in the general case where the element may require additional alignment.

[AaronRobinsonMSFT]

it may matter in the general case where the element may require additional alignment.

That would seem to imply the generator needs to know not only size, but also alignment. I think you're advocating for a BufferAlignment field on the CustomTypeMarshaller type?

[jkotas]

The algorithm to compute alignment requirements from a type tends to be platform and architecture specific. I do not think we would want to include it in the source generator. The best way to specify alignment requirement is by specifying the type.

[AaronRobinsonMSFT]

The algorithm to compute alignment requirements from a type tends to be platform and architecture specific. I do not think we would want to include it in the source generator.

Agreed, that is what I wanted to clarify.

The best way to specify alignment requirement is by specifying the type.

I think this will work on some primitive types, but I don't know if it will on more complex types. For example, a value type with various fields - is the intent here to defer to the JIT to ensure the correct alignment for the specific platform?

[jkotas]

is the intent here to defer to the JIT to ensure the correct alignment for the specific platform?

Yes.

[jaredpar]

I mentioned it to @jaredpar a while back the last time it came up on the C# discord. Don't remember what his response though.

The ability to support pointers in generic arguments at the language level I would classify as "mostly understood". Particularly because it's likely to just be a small tweak on how we support ref struct as generic arguments. It's an incremental cost at that point.

For me it's more a question of value and if it's worth it. This requires runtime and language changes so have to make sure it's valued across the stack compared to everything else we're doing. I don't feel like I have the best context to make that decision. The instances I've seen where this would be useful have been fairly low. But I also don't get pinged on that area a lot so could be an exposure issue.

I don't see it fitting in .NET 7 though given other features we've pushed out

[elinor-fung]

Should the buffer be strongly typed (ie Span) to communicate the required alignment? It probably does not matter a whole lot for this specific case, but it may matter in the general case where the element may require additional alignment.

For the marshaller shapes in #46838, I believe we went with the shape of the constructor having Span<byte> (and CustomTypeMarshaller.BufferSize in bytes) since that could be in common for any marshaller. Is the argument then that it should be Span of any unmanaged type? cc @jkoritzinsky

[jkoritzinsky]

I think supporting Span<T> for an unmanaged T to guarantee alignment requirements is interesting. However, I don't believe it would be able to cover all possible scenarios. In particular, I don't know if we could implement a mechanism to ensure alignment would be correct for arrays with custom element marshalling with this mechanism. I'll try to think of a design that would work for that case as well.

[jkoritzinsky]

After talking it over with @elinor-fung, here's what we decided:

• Enabling users to provide Span<T> for any unmanaged T for the buffer to guarantee alignment requirements is a good idea and we should do it.
• Guaranteeing alignment requirements for LinearCollection marshallers is more difficult, but we could guarantee stack alignment is at least good enough for the native element type if we decide to do so.
  • We don't have a mechanism to provide the alignment requirements to the marshaller for dynamic allocations though, so it might not be worthwhile to guarantee stack allocation alignment today. Also, we have never guaranteed stack alignment for elements of array marshalling, so we probably don't need to do that now and can leave that for developers to do for their scenario if required.

[elinor-fung]

Updated to Span<ushort> for Utf16StringMarshaller.

[jkotas]

Updated to Span for Utf16StringMarshaller.

Should the buffer size be changed to 0x100 as well? (I assume that the buffer size is in sizeof(ushort) now.)

[elinor-fung]

Oops, fixed.

[bartonjs]

Video

• We've generally stopped using "Ptr" and instead use "Pointer", so PtrArrayMarshaller got renamed to PointerArrayMarshaller
• Consider if it's valuable to put these (and future) marshallers in a sub-namespace, e.g. System.Runtime.InteropServices.Marshallers
  • If it seems valueable to also move the new LibraryImport attribute, something slightly more general, like System.Runtime.Marshalling for "the new marshalling" namespace

namespace System.Runtime.InteropServices
{
    [CLSCompliant(false)]
    [CustomTypeMarshaller(typeof(string), BufferSize = 0x100,
        Features = CustomTypeMarshallerFeatures.UnmanagedResources | CustomTypeMarshallerFeatures.CallerAllocatedBuffer | CustomTypeMarshallerFeatures.TwoStageMarshalling )]
    public unsafe ref struct AnsiStringMarshaller
    {
        public AnsiStringMarshaller(string? str);
        public AnsiStringMarshaller(string? str, Span<byte> buffer);

        public byte* ToNativeValue();
        public void FromNativeValue(byte* value);

        public string? ToManaged();

        public void FreeNative();
    }

    [CLSCompliant(false)]
    [CustomTypeMarshaller(typeof(string), BufferSize = 0x100,
        Features = CustomTypeMarshallerFeatures.UnmanagedResources | CustomTypeMarshallerFeatures.CallerAllocatedBuffer | CustomTypeMarshallerFeatures.TwoStageMarshalling )]
    public unsafe ref struct Utf8StringMarshaller
    {
        public Utf8StringMarshaller(string? str);
        public Utf8StringMarshaller(string? str, Span<byte> buffer);

        public byte* ToNativeValue();
        public void FromNativeValue(byte* value);

        public string? ToManaged();

        public void FreeNative();
    }

    [CLSCompliant(false)]
    [CustomTypeMarshaller(typeof(string), BufferSize = 0x100,
        Features = CustomTypeMarshallerFeatures.UnmanagedResources | CustomTypeMarshallerFeatures.CallerAllocatedBuffer | CustomTypeMarshallerFeatures.TwoStageMarshalling)]
    public unsafe ref struct Utf16StringMarshaller
    {
        public Utf16StringMarshaller(string? str);
        public Utf16StringMarshaller(string? str, Span<ushort> buffer);

        public ref ushort GetPinnableReference();

        public ushort* ToNativeValue();
        public void FromNativeValue(ushort* value);

        public string? ToManaged();

        public void FreeNative();
    }

    [CustomTypeMarshaller(typeof(CustomTypeMarshallerAttribute.GenericPlaceholder[]),
        CustomTypeMarshallerKind.LinearCollection, BufferSize = 0x200,
        Features = CustomTypeMarshallerFeatures.UnmanagedResources | CustomTypeMarshallerFeatures.CallerAllocatedBuffer | CustomTypeMarshallerFeatures.TwoStageMarshalling)]
    public unsafe ref struct ArrayMarshaller<T>
    {
        public ArrayMarshaller(int sizeOfNativeElement);
        public ArrayMarshaller(T[]? array, int sizeOfNativeElement);
        public ArrayMarshaller(T[]? array, Span<byte> buffer, int sizeOfNativeElement);

        public ReadOnlySpan<T> GetManagedValuesSource();

        public Span<T> GetManagedValuesDestination(int length);
        public Span<byte> GetNativeValuesDestination();

        public ReadOnlySpan<byte> GetNativeValuesSource(int length);

        public ref byte GetPinnableReference();

        public byte* ToNativeValue();
        public void FromNativeValue(byte* value);

        public T[]? ToManaged();

        public void FreeNative();
    }

    [CustomTypeMarshaller(typeof(CustomTypeMarshallerAttribute.GenericPlaceholder*[]),
        CustomTypeMarshallerKind.LinearCollection, BufferSize = 0x200,
        Features = CustomTypeMarshallerFeatures.UnmanagedResources | CustomTypeMarshallerFeatures.CallerAllocatedBuffer | CustomTypeMarshallerFeatures.TwoStageMarshalling)]
    public unsafe ref struct PointerArrayMarshaller<T> where T : unmanaged
    {
        public PointerArrayMarshaller(int sizeOfNativeElement);
        public PointerArrayMarshaller(T*[]? array, int sizeOfNativeElement);
        public PointerArrayMarshaller(T*[]? array, Span<byte> buffer, int sizeOfNativeElement);

        public ReadOnlySpan<IntPtr> GetManagedValuesSource();

        public Span<IntPtr> GetManagedValuesDestination(int length);
        public Span<byte> GetNativeValuesDestination();

        public ReadOnlySpan<byte> GetNativeValuesSource(int length);
        
        public ref byte GetPinnableReference();

        public byte* ToNativeValue();
        public void FromNativeValue(byte* value);

        public T*[]? ToManaged();

        public void FreeNative();
    }
}