Add struct marshalling design doc.

DllImportGenerator/designs/StructMarshalling.md

@@ -0,0 +1,127 @@
+# Struct Marshalling
+
+As part of the new source-generated direction for .NET Interop, we are looking at various options for supporting marshaling user-defined struct types.
+
+These types pose an interesting problem for a number of reasons listed below. With a few constraints, I believe we can create a system that will enable users to use their own user-defined types and pass them by-value to native code.
+
+## Problems
+
+- Unmanaged vs Blittable
+  - The C# language (and Roslyn) do not have a concept of "blittable types". It only has the concept of "unmanaged types", which is similar to blittable, but differs for `bool`s and `char`s. `bool` and `char` types are "unmanaged", but are never (in the case of `bool`), or only sometimes (in the case of `char`) blittable. As a result, we cannot use the "is this type unmanaged" check in Roslyn for structures.
+- Limited type information in ref assemblies.
+  - In the ref assemblies generated by dotnet/runtime, we save space and prevent users from relying on private implementation details of structures by emitting limited information about their fields. Structures that have at least one non-object field are given a private `int` field, and structures that have at least one field that transitively contains an object are given one private `object`-typed field. As a result, we do not have full type information at code-generation time for any structures defined in the BCL when compiling a library that uses the ref assemblies.
+- Private reflection
+  - Even when we do have information about all of the fields, we can't emit code that references them if they are private, so we would have to emit unsafe code and calculate offsets manually to support marshaling them.
+
+## Possible Solution A - Opt-In Structure Interop
+
+We've been working around another problem for a while in the runtime-integrated interop design: The user can use any type that is non-auto layout and has fields that can be marshaled in interop. This has lead to various issues where types that were not intended for interop usage became usable and then we couldn't update their behavior to be special-cased since users may have been relying on the generic behavior (`Span<T>`, `Vector<T>` come to mind).
+
+I propose an opt-in design where the owner of a struct has to explicitly opt-in to usage for interop. This enables our team to add special support as desired for various types such as `Span<T>` while also avoiding the private reflection and limited type information issues mentioned above.
+
+This design would use three attributes:
+
+```csharp
+
+public class GeneratedMarshallingAttribute : Attribute {}
+
+public class BlittableTypeAttribute : Attribute {}
+
+public class NativeStructMarshallingAttribute : Attribute
+{
+     public NativeStructMarshallingAttribute(Type nativeStructType, string managedToNativeMarshalMethod, string nativeToManagedMarshalMethod, string nativeFreeMethod) {}
+}
+
+```
+
+There are 2 usage mechanisms of these attributes.
+
+- Usage 1, Source-generated interop:
+
+The user can apply the `GeneratedMarshallingAttribute` to their structure `S`. The source generator will determine if the type is blittable. If it is blittable, the source generator will generate a partial definition and apply the `BlittableTypeAttribute` to the struct type `S`. Otherwise, it will generate a blittable representation of the struct and apply the `NativeStructMarshallingAttribute` and point it to the blittable representation, a method that marshals the managed value to the native value, a method that marshals a value of the native struct to a managed value, and a method that frees any native resources that were allocated for the native value. The blittable representation can either be generated as a separate top-level type or as a nested type on `S`.
+
+- Usage 2, Manual interop:
+
+The user may want to manually mark their types as marshalable in this system due to specific restrictions in their code base around marshaling specific types that the source generator does not account for. We could also use this internally to support custom types in source instead of in the code generator. In this scenario, the user would apply either the `BlittableTypeAttribute` or the `NativeStructMarshallingAttribute` attribute to their struct type. An analyzer would validate that the struct is blittable if the `BlittableTypeAttribute` is applied or validate that the native struct type is blittable and that the methods provided in the attribute have the correct signatures when the `NativeStructMarshallingAttribute` is applied.
+
+The P/Invoke source generator (as well as the struct source generator when nested struct types are used) would use the `BlittableTypeAttribute` and `NativeStructMarshallingAttribute` to determine how to marshal a value type parameter or field instead of looking at the fields of the struct directly.
+
+The largest problem of this design is the switch to an opt-in model for structure types. We could augment this design by providing a fallback to the built-in struct marshaling system via an attribute (either `MarshalAsAttribute` or another one) if we believe providing an explicit escape hatch is needed.
+
+#### Why do we need `BlittableTypeAttribute`?
+
+Based on the design above, it seems that we wouldn't need `BlittableTypeAttribute`. However, due to the ref assembly issue above in combination with the desire to enable manual interop, we need to provide a way for users to signal that a given type should be blittable and that the source generator should not generate marshalling code.
+
+I'll give a specific example for where we need the `BlittableTypeAttribute` below. Let's take a scenario where we don't have `BlittableTypeAttribute`.
+
+In Foo.csproj, we have the following types:
+
+```csharp
+public struct Foo
+{
+     private bool b;
+}
+
+public struct Bar
+{
+     private short s;
+}
+```
+
+We compile these types into an assembly Foo.dll and we want to publish a package. We decide to use infrastructure similar to dotnet/runtime and produce a ref assembly. The ref assembly will have the following types:
+
+```csharp
+struct Foo
+{
+     private int dummy;
+}
+struct Bar
+{
+     private int dummy;
+}
+```
+
+We package up the ref and impl assemblies and ship them in a NuGet package.
+
+Someone else pulls down this package and writes their own struct type Baz1 and Baz2:
+
+```csharp
+struct Baz1
+{
+     private Foo f;
+}
+struct Baz2
+{
+     private Bar b;
+}
+```
+
+Since the source generator only sees ref assemblies, it would think that both `Baz1` and `Baz2` are blittable, when in reality only `Baz2` is blittable. This is the ref assembly issue mentioned above. The source generator cannot trust the shape of structures in other assemblies since those types may have private implementation details hidden.
+
+Now let's take this scenario again with `BlittableTypeAttribute`:
+
+```csharp
+[BlittableType]
+public struct Foo
+{
+     private bool b;
+}
+
+[BlittableType]
+public struct Bar
+{
+     private short s;
+}
+```
+
+This time, we produce an error since Foo is not blittable. We need to either apply the `GeneratedMarshalling` attribute (to generate marshalling code) or the `NativeStructMarshalling` attribute (so provide manually written marshalling code) to Foo. This is also why we require each type used in interop to have either a `[BlittableType]` attribute or a `[NativeStructMarshalling]` attribute; we can't validate the shape of a type not defined in the current assembly because its shape may be different between its reference assembly and the runtime type.
+
+Now there's another question: Why we can't just say that a type with `[GeneratedMarshalling]` and not `[NativeStructMarshalling]` has been considered blittable?
+
+We don't want to require usage of `[GeneratedMarshalling]` to mark that a type is blittable because then there is no way to enforce that the type is blittable. If we require usage of `[GeneratedMarshalling]`, then we will automatically generate marshalling code if the type is not blittable. By also having the `[BlittableType]` attribute, we enable users to mark types that they want to ensure are blittable and an analyzer will validate the blittability.
+
+Basically, the design of this feature is as follows:
+
+At build time, the user can apply either `[GeneratedMarshallling]`, `[BlittableType]`, or `[NativeStructMarshalling]`. If they apply `[GeneratedMarshalling]`, then the source generator will run and generate marshalling code as needed and apply either `[BlittableType]` or `[NativeStructMarshalling]`. If the user manually applies `[BlittableType]` or `[NativeStructMarshalling]` instead of `[GeneratedMarshalling]`, then an analyzer validates that the type is blittable (for `[BlitttableType]`) or that the marshalling methods and types have the required shapes (for `[NativeStructMarshalling]`).
+
+When the source generator (either Struct, P/Invoke, Reverse P/Invoke, etc.) encounters a struct type, it will look for either the `[BlittableType]` or the `[NativeStructMarshalling]` attributes to determine how to marshal the structure. If neither of these attributes are applied, then the struct cannot be passed by value.

DllImportGenerator/readme.md

@@ -13,4 +13,8 @@ Below are additional work items that are not presently captured in this reposito
 
 ### Optional
 
-* A tool to compare the resulting IL from the generated source to that generated by the built-in IL Marshaller system. This would help with validation of what is being generated.
+* A tool to compare the resulting IL from the generated source to that generated by the built-in IL Marshaller system. This would help with validation of what is being generated.
+
+## Designs
+
+- [Struct Marshalling](./designs/StructMarshalling.md)