Proposal: For static extern methods, C# should support passing null to any out or ref parameters marked with [Optional].

[tannergooding]

Ported from dotnet/roslyn#13119

Issue
There are a number of interop scenarios where a static extern method takes an out parameter and allows it to be null.

For Example

HRESULT WINAPI D3D12CreateDevice(
  _In_opt_  IUnknown          *pAdapter,
            D3D_FEATURE_LEVEL MinimumFeatureLevel,
  _In_      REFIID            riid,
  _Out_opt_ void              **ppDevice
);

might translate to

[DllImport("D3D12.dll", BestFitMapping = false, CallingConvention = CallingConvention.Winapi, CharSet = CharSet.Unicode, EntryPoint = "D3D12CreateDevice", ExactSpelling = true, PreserveSig = true, SetLastError = false, ThrowOnUnmappableChar = false)]
public static extern int CreateDevice(
    [In, Optional] IntPtr pAdapter,
    [In] uint MinimumFeatureLevel,
    [In] ref Guid riid,
    [Out, Optional] out IntPtr ppDevice
);

However, C# does not currently support passing null to the optional fourth parameter.

Workaround
Use unsafe code and remove the out keyword so that the user can actually pass null.

For example, the above becomes:

[DllImport("D3D12.dll", BestFitMapping = false, CallingConvention = CallingConvention.Winapi, CharSet = CharSet.Unicode, EntryPoint = "D3D12CreateDevice", ExactSpelling = true, PreserveSig = true, SetLastError = false, ThrowOnUnmappableChar = false)]
unsafe public static extern int CreateDevice(
    [In, Optional] IntPtr pAdapter,
    [In] uint MinimumFeatureLevel,
    [In] ref Guid riid,
    [Out, Optional] IntPtr* ppDevice
);

Proposal
For static extern methods, C# should support passing null to any out or ref parameter marked with Optional.

This would allow you to write code such as:

[DllImport("D3D12.dll", BestFitMapping = false, CallingConvention = CallingConvention.Winapi, CharSet = CharSet.Unicode, EntryPoint = "D3D12CreateDevice", ExactSpelling = true, PreserveSig = true, SetLastError = false, ThrowOnUnmappableChar = false)]
public static extern int CreateDevice(
    [In, Optional] IntPtr pAdapter,
    [In] uint MinimumFeatureLevel,
    [In] ref Guid riid,
    [Out, Optional] out IntPtr ppDevice
);

public static int CreateDevice(IntPtr pAdapter, uint MinimumFeatureLevel, ref Guid riid)
{
    return CreateDevice(pAdapter, MinimumFeatureLevel, ref riid, null); // or possibly: out null
}

Other Thoughts
It would be immensely useful if this was supported in other scenarios as well, such as for COM Interop.

For example, one might modify the above function to take a strongly type interface:

[Guid("2411E7E1-12AC-4CCF-BD14-9798E8534DC0"), InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
unsafe public interface IDXGIAdapter : IDXGIObject
{
    // Method Declarations
}

[Guid("189819F1-1DB6-4B57-BE54-1821339B85F7"), InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
unsafe public interface ID3D12Device : ID3D12Object
{
    // Method Declarations

    [PreserveSig]
    HRESULT CreateHeap(
        [In] ref D3D12_HEAP_DESC pDesc,
        [In] ref Guid riid,
        [Out, Optional] out ID3D12Heap ppvHeap
    );

    // More Method Declarations
}

[DllImport("D3D12.dll", BestFitMapping = false, CallingConvention = CallingConvention.Winapi, CharSet = CharSet.Unicode, EntryPoint = "D3D12CreateDevice", ExactSpelling = true, PreserveSig = true, SetLastError = false, ThrowOnUnmappableChar = false)]
unsafe public static extern int CreateDevice(
    [In, Optional] IDXGIAdapter pAdapter,
    [In] uint MinimumFeatureLevel,
    [In] ref Guid riid,
    [Out, Optional] out ID3D12Device ppDevice
);

The proposal would work for the static extern method, but it would not work for the ID3D12Device.CreateHeap method.

Support for the static extern method is much simpler, because the code is guaranteed to not exist in managed code.

However, support for the latter would require some extension to the language. Such an extension would either indicate the interface/delegate/method exist in native code (and can therefore handle a null out parameter) or it would allow the language itself to work with null out parameters (currently there is no way to determine that the out is null, only that the object out points to is null).

It should be noted that [ComImport] does provide some of the functionality desired for optional out parameters on interfaces. However, applying the ComImport attribute may provide functionality that is undesirable in some COM interop scenarios.

[jnm2]

This would allow me to reduce the number of pinvoke overloads I maintain. (@VSadov nullable ref parameters?)

[VSadov]

 [Out, Optional] IntPtr* ppDevice

seems the very right way to handle this situation. The callee expects a pointer and has a special handling of nullptr. That is why unsafe code exists - to handle such situations when doing interop with native code.

Mapping such parameter to an out parameter would indeed cause problems. What kind of argument-passing convention is this? - Are you passing the argument expression by value or by reference?
Is it "it depends" calling convention?

The proposal tries to special case certain methods and special case certain arguments.
That will have to be considered in a lot of places where method calls are involved - delegates, overload resolution, lambda inference and so on.

Basically - I understand the issue, but do not see it as deserving this kind of solution.

[jnm2]

Basically - I understand the issue, but do not see it as deserving this kind of solution.

Unless it fits into a wider picture. Suppose nullable refs become an ordinary managed thing as well, as has been requested?

[tannergooding]

I think this now has more precedence with the existence of ref returns and ref readonly. With these concepts now being supported in managed code, APIs that use ref/out will likely become much more prevalent.

Having this support allows the user to not allocate when it is unnecessary (even if said allocation is only on the stack). It also reduces the API surface area for interop code (no longer have to have a 'safe' version for normal use and an 'unsafe' version just for when doing validation of input parameters).

[tannergooding]

@jnm2, we can workaround this right now using System.Runtime.CompilerServices.Unsafe.AsRef<T>(null) 😄

[jnm2]

@tannergooding When's that coming to .NET Framework 😄

[tannergooding]

@jnm2, it is available already. System.Runtime.CompilerServices.Unsafe is an independent NuGet package that works on both Desktop and Core.

[radobot]

I've stumbled upon this problem too when I was writing code for the Vulkan C++ API. In it there is the function

VkResult vkEnumerateInstanceLayerProperties(
	uint32_t*          pPropertyCount,
	VkLayerProperties* pProperties
);

which returns either the number of layers in the first parameter or the layers themselves (by writing in an array) in the second parameter depending on whether the second parameter is null or not.

I think fundamentally the problem here is that one function is being used for two different behaviours. This works for native code because there is a lot of freedom in what value can be shoved into what variable or argument, but in C# there is a lot less freedom - that's kind of the whole point of C#, it tries to be safer.

Therefore I think this can be solved in one of two ways:

• Make C# behave closer to native code -> use unsafe.
  In my example it would be

[DllImport(vulkanLibrary, CallingConvention = CallingConvention.Winapi, EntryPoint = "vkEnumerateInstanceLayerProperties", ExactSpelling = true)]
private static extern unsafe Result EnumerateInstanceLayerProperties(
	ref UInt32       propertyCount,
	LayerProperties* properties
);

which enables both behaviours but requires the use of unsafe code.

Of course there's also the option of using Marshalling, but that's not really viable for performance critical code. (For example the Vulkan API contains many other functions that behave like this one.)

• Support only one behaviour at a time -> use method overloading.
  Which in this case would look like

[DllImport(vulkanLibrary, CallingConvention = CallingConvention.Winapi, EntryPoint = "vkEnumerateInstanceLayerProperties", ExactSpelling = true)]
private static extern Result EnumerateInstanceLayerProperties(
	out UInt32 propertyCount,
	UIntPtr    properties /* = UIntPtr.Zero */
);

[DllImport(vulkanLibrary, CallingConvention = CallingConvention.Winapi, EntryPoint = "vkEnumerateInstanceLayerProperties", ExactSpelling = true)]
private static extern Result EnumerateInstanceLayerProperties(
	in UInt32               propertyCount,
	[Out] LayerProperties[] properties
);

and then I can do

public static Result EnumerateInstanceLayerProperties(
	out UInt32 propertyCount
) => EnumerateInstanceLayerProperties(
	out propertyCount,
	UIntPtr.Zero
);

public static Result EnumerateInstanceLayerProperties(
	LayerProperties[] properties
) => EnumerateInstanceLayerProperties(
	(uint)properties.Length,
	properties
);

which I think would be the preferred way of doing these kinds of things.

Also there's the option of using nullable types - using classes instead of structures, which solves some of the simpler types of these kinds of problems. (Especially if used together with in/out/ref to create double pointers as explained in https://docs.microsoft.com/en-us/dotnet/framework/interop/passing-structures)

In conclusion I think that the interoperability capabilities of C# are quite powerful already and I don't think it would be possible to (even theoretically) seamlessly interoperate between native and managed code without either breaking the safety benefits of using managed code or introducing (possibly large) overhead.

[mpolicki]

I often find myself working with the Win32 API and I come across functions with out or ref parameters that also accept NULL as valid input. It would be great to not have to write workarounds or have unsafe code, and instead just be able to pass null.

[333fred]

Note that since this was opened, C# has added support for out discards, out _. We explicitly did not add similar support for ref null as that seems like a very easy way to shoot yourself in the foot and then have a hard time debugging.

[tannergooding]

For P/Invoke in particular, out discards are not a viable option. There is a distinct difference between out Unsafe.NullRef() and out _ in that the latter will not pass null but a valid pointer to a hidden stack object of the right type instead. This will impact what native code sees and will ultimately not cause it to go down the right code path.

[atynagano]

I recently faced the same problem. I don't know how many people are looking at this thread right now, but I'd like to share my perspective.

There are many functions in the Win32 API that have an optional out parameter.
Using Unsafe.NullRef<T>() is certainly a solution, but it is somewhat too long to type and the signature of the function does not allow me to determine if I can use Unsafe.NullRef<T>() or not.

Another solution is to have an overload (or a function with a different name) that omits optional out parameter. However, there is a disadvantage: if there is more than one optional out parameter, then 2^n overloads are needed. This is not reasonable.

I think we need a new attribute like the one below.

D3D12CreateDevice(
      IUnknown? pAdapter,
      D3D_FEATURE_LEVEL MinimumFeatureLevel,
      Guid* riid,
      [OptionalOut] out IntPtr ppDevice);

The [OptionalOut] attribute allow a caller to generate Unsafe.NullRef<T>() instead of generating a hidden local variable if the caller use out _.
For Example, The following code

D3D12CreateDevice(adapter, level, riid, out _);

is evaluated as

D3D12CreateDevice(adapter, level, riid, out Unsafe.NullRef<IntPtr>());

In this way, a caller can use out _ without worrying about the implementation.
A callee can safely deal with a null reference by pinning or using Unsafe.IsNullRef(). I believe the compiler can do this without breaking changes.

P.S.
I realized that if a caller and a callee are located in different libraries, a problem may occurs when the callee removes the attribute.

[atynagano]

I realized that ref field may solve the problem.
csharplang/low-level-struct-improvements.md at main · dotnet/csharplang

readonly ref struct Option<T>
{
	private ref T _ref;

	public Option(out T value)
	{
		Unsafe.SkipInit(out value);
		_ref = ref value;
	}

	public static Option<T> Null => default;

	public ref T GetPinnableReference() => ref _ref;
}

void Caller()
{
	Callee(Option<int>.Null);
	Callee(new Option<int>(out var i));
}

void Callee(Option<int> option = default)
{
	unsafe {
		fixed(int* ptr = option) {
			// do something with ptr
		}
	}
}