Improving and unifying handling of native Libraries code in embedded/singlefile scenarios

[VSadov]

Improving handling of native Libraries code in embedded/singlefile scenarios

For reasons that vary .Net libraries rely on nontrivial amount of native code. The native code is factored into a few well-known native dynamic-link libraries (.dll, .so, etc). As an example we have 4 libraries like this on Linux – "libSystem.Native", "libSystem.IO.Compression.Native", "libSystem.Net.Security.Native", "libSystem.Security.Cryptography.Native.OpenSsl". The number of libraries varies depending on platform.

This is a very simple model which has its own advantages. In particular there is no need for coreclr to have any special knowledge about these libraries.

Where this becomes inconvenient and fragile is when we want to package the native libraries differently - for example as a part of a single file host. To handle such requirement in the current implementation we produce static libraries for the native code and link that into coreclr/host. Since the native dlls are not longer around in this configuration, coreclr intercepts loading of these libraries and redirects to the containing host binary.

There is a number of issues:

• Since coreclr and libraries are built separately, there is a possibility of a mismatch that could make them incompatible. (i.e. Debug vs. Release, different versions of libc, new/free operator mismatch ...)
• Intercepting and redirecting the load happens in PAL layer. This is a problem when the same must be done on Windows.
• Accessing the native methods via regular PInvokes requires that the native host re-exports the methods. That easily runs into platform or linker dependent behaviors regarding re-exporting of library functions from final executables.

Proposal:

• Build native libraries as a part of CoreClr build.
  The source files should still be located under Libraries partition, so that they can be shared in source form with Mono, for example.

• The “embedded” case builds native libraries as a part of coreclr binary, thus bypassing the explicit lib files.

• The "embedded" case recognizes well-known libraries as a special case of QCall and dispatches calls to the embedded implementations.
  Example: “libSystem.Native” would normally be a name of the native library to load, but in “embedded” case, the DLLEXPORT methods of libSystem.Native would be registered in the runtime similarly to QCall and “libSystem.Native” would be recognized as another QCall variety.
  QCall mechanism here allows us to “shim” the location of native methods while not changing the managed code that calls them.

• The QCall registration table for the native calls will be automatically generated by scanning managed libraries.

Some unresolved details:

• If managed callers are built after coreclr, we would need to reconcile how to do the QCall generation before coreclr builds.
• Should the generation of QCall entries be a part of the build (producing a temporary source file) or a dev-time step (the file is explicitly generated and stored in Git).

[Dotnet-GitSync-Bot]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[trylek]

Assigning to .NET 6 milestone as this is a design proposal, not a bug fix, and I believe it's too late in the .NET 5 release cycle to introduce new feature work.

[VSadov]

This was implemented in #44505