AssemblyDependencyResolver raises exception when hosting .NET using using Mscoree.h

[cheverdyukv]

Description

Creating instance of AssemblyDependencyResolver throws exception "Hostpolicy must be initialized and corehost_main must have been called before calling corehost_resolve_component_dependencies" when native host uses Mscoree.h to load .NET 5 runtime.

Steps:

1. Create host using https://docs.microsoft.com/en-us/dotnet/core/tutorials/netcore-hosting#create-a-host-using-mscoreeh as example.
2. Run managed code using CreateDelegate approach.
3. In .NET code create instance of AssemblyDependencyResolver.

Expected behavior: I believe during host initialization I provided enough information to allow AssemblyDependencyResolver work
Actual behavior: Exception is raised in corehost_resolve_component_dependencies

Configuration

• .NET preview 5, but I don't think there were any changes in preview 6.
• Windows 10
• Using x64 but I don't think it is specific to architecture

[Dotnet-GitSync-Bot]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[ghost]

Tagging subscribers to this area: @vitek-karas, @swaroop-sridhar, @agocke
Notify danmosemsft if you want to be subscribed.

[vitek-karas]

This is currently by design. The AssemblyDependencyResolver currently only works if the runtime is hosted via hostfxr (and hostpolicy).
Please follow the guide here: https://docs.microsoft.com/en-us/dotnet/core/tutorials/netcore-hosting#create-a-host-using-nethosth-and-hostfxrh, to host the runtime from your native code.

[cheverdyukv]

Well I cannot use that option, because I'm using non default appDomainFlags and there is no way to provide them using hostpolicy.

[vitek-karas]

What non-default flags do you use (and if you can provide that info - why)? I want to figure out if there's a scenario which hostpolicy approach should cover in the future.

[cheverdyukv]

1. APPDOMAIN_ENABLE_PINVOKE_AND_CLASSIC_COMINTEROP because we are using COM. I am not sure if it is default flag
2. APPDOMAIN_IGNORE_UNHANDLED_EXCEPTIONS because we use some 3rd party components that creates threads and raises unhandled exception. And in general we process all unhandled exceptions anyway using appropriate callbacks, we just don't need runtime to terminate process.
3. "Create a host using NetHost.h and HostFxr.h" expects config path and in many cases we are loading .NET Framework assembly and there is no config for it. Our application eventually loading around 200 assemblies and planning to port them to .NET Core. But it takes time and until we finish a lot of them will be in .NET Framework.
4. "Create a host using Mscoree.h" allows to specify where are .NET Core assembly located. We are planning to ship .NET Core with our app. I could be wrong but I don't think it is possible to do the same using "Create a host using NetHost.h and HostFxr.h"
5. I don't think "Create a host using NetHost.h and HostFxr.h" allows to specify APP_PATHS, NATIVE_DLL_SEARCH_DIRECTORIES etc that we are actively using. We have a lot of 3rd party components and they are located in different directories.
6. We also using start up flags STARTUP_CONCURRENT_GC, STARTUP_SINGLE_APPDOMAIN STARTUP_LOADER_OPTIMIZATION_SINGLE_DOMAIN and I also not sure how to pass them in "Create a host using NetHost.h and HostFxr.h"

[vitek-karas]

I'll try to answer as I investigate:

1. APPDOMAIN_ENABLE_PINVOKE_AND_CLASSIC_COMINTEROP because we are using COM. I am not sure if it is default flag

This is on by default when using the new initialization APIs (which hostfxr/hostpolicy does): https://github.com/dotnet/runtime/blob/master/src/coreclr/src/dlls/mscoree/unixinterface.cpp#L243

2. APPDOMAIN_IGNORE_UNHANDLED_EXCEPTIONS because we use some 3rd party components that creates threads and raises unhandled exception. And in general we process all unhandled exceptions anyway using appropriate callbacks, we just don't need runtime to terminate process.

I don't know the answer to this - @janvorli would you know if there's a way to get this behavior without setting the flag on AppDomain via native code?

3. "Create a host using NetHost.h and HostFxr.h" expects config path and in many cases we are loading .NET Framework assembly and there is no config for it. Our application eventually loading around 200 assemblies and planning to port them to .NET Core. But it takes time and until we finish a lot of them will be in .NET Framework.
4. "Create a host using Mscoree.h" allows to specify where are .NET Core assembly located. We are planning to ship .NET Core with our app. I could be wrong but I don't think it is possible to do the same using "Create a host using NetHost.h and HostFxr.h"

The config path is just a way to find the .NET Core runtime, you are doing this manually now. If you need to use self-contained runtime (meaning you ship the runtime with the app) - this is supported for application in 3.1, but if you need to load the managed code as "components" into otherwise native app then something similar was added in .NET 5. See #35465 for a detailed discussion on a very similar requirement. The new functionality introduced as part of this issue should be a solution for you - see https://github.com/dotnet/runtime/blob/master/docs/design/features/native-hosting.md#calling-managed-function-net-5-and-above for the new API description. Note that with self-contained apps the config file is effectively optional anyway.

5. I don't think "Create a host using NetHost.h and HostFxr.h" allows to specify APP_PATHS, NATIVE_DLL_SEARCH_DIRECTORIES etc that we are actively using. We have a lot of 3rd party components and they are located in different directories.

It is possible - https://github.com/dotnet/runtime/blob/master/docs/design/features/native-hosting.md#inspect-and-modify-host-context - you can call hostfxr_set_runtime_property_value to overwrite/add any of the runtime properties this way.

6. We also using start up flags STARTUP_CONCURRENT_GC, STARTUP_SINGLE_APPDOMAIN STARTUP_LOADER_OPTIMIZATION_SINGLE_DOMAIN and I also not sure how to pass them in "Create a host using NetHost.h and HostFxr.h"

STARTUP_LOADER_OPTIMIZATION_SINGLE_DOMAIN and STARTUP_SINGLE_APPDOMAIN are set by default: https://github.com/dotnet/runtime/blob/master/src/coreclr/src/dlls/mscoree/unixinterface.cpp#L93

STARTUP_CONCURRENT_GC can be specified via a runtime property System.GC.Concurrent - this can be set either from the .runtimeconfig.json or by calling the hostfxr_set_runtime_property_value.

[cheverdyukv]

Thank you so much for such detailed answer!

2. I will wait for answer on this

Everything else looks like working correctly. Except that I got strange errors everywhere in WPF that I didn't have before. Could it be related to hdt_load_assembly_and_get_function_pointer function that loading assemblies into secondary ALC?

If yes, is there any other way to load assembly into default ALC by specifying assembly file name? I did read about hdt_get_function_pointer but there is no way to specify assembly path. Our application has many plugins and they are in different directories and adding them to APP_PATHS is huge overkill. And I believe it is not possible to modify APP_PATHS after runtime is initialized but our application allows to download plugin while application is running.

[vitek-karas]

Running WPF in secondary ALC is known to have issues - some of it might be possible to overcome with https://docs.microsoft.com/en-us/dotnet/api/system.runtime.loader.assemblyloadcontext.currentcontextualreflectioncontext?view=netcore-3.1 but I don't know if it solves all known problems.

We don't have a first class support for loading assemblies by path into default ALC - simply didn't get to it yet (and it's also a bit tricky as using AssemblyDependencyResolver with default ALC requires different approach than the one used in secondary ALC). That said you can do this yourself:

• If you want to use self-contained deployment you will need to go the hostfxr_initialize_for_dotnet_commandline route anyway as that's the only supported way to load self-contained code view the native hosting APIs (true outside of these APIs as well for now).
• So you will need to have at least some small application acting as a bootstrapper for the .NET Core code
  • This will declare dependencies on all the frameworks you need (and which versions) - effectively it will deliver the .runtimeconfig.json.
  • It will contain the default set of assemblies to be loaded into default ALC
  • It can have empty Main as you're not going to call that method ever
• With that you can relatively easily write a managed method in this bootstrapper app to facilitate the loading. hdt_load_assembly_and_get_function_pointer is basically equivalent to calling hdt_get_function_pointer on ComponentActivator.LoadAssemblyAndGetFunctionPointer and then calling the returned function pointer (effectively that method) to load the assembly into secondary ALC and return the custom managed function. You can basically implement your own load method and get it via hdt_get_function_pointer and then call it to load the assembly and get some function from it - and you can load the assembly into the default ALC whichever way you need.

I know this is not exactly nice, but currently it's the only solution I can think of (until there's direct support for this in the .NET Core).

[cheverdyukv]

I created new assembly and it has only one class, one method and one delegate. Then I got pointer to that method via hdt_load_assembly_and_get_function_pointer. There I have following code:

public static /*HRESULT*/int Create([MarshalAs(UnmanagedType.Interface)] out object result)
{
  ...
  Assembly assembly = AssemblyLoadContext.Default.LoadFromAssemblyPath(dotNetSupportFileName);
  Type t = assembly.GetType("DotNetSupport.DotNetTools", throwOnError: true, false);
  object o = Activator.CreateInstance(t, BindingFlags.Instance | BindingFlags.Public | BindingFlags.CreateInstance, null, null, null, null);
  ...

and in DotNetTools I have

public object CreateObject(string assemblyFileName, string typeName)
{
  return Activator.CreateInstanceFrom(assemblyFileName, typeName);
}

that will load assembly from path and create root object for our application to use. I seems to work and WPF works correctly.

1. Is it correct way to do what you said before?

2. I create directory Runtime\shared and copied Microsoft.NETCore.App Microsoft.WindowsDesktop.App to that location. Then I assigned that FullPathToAppDir\Runtime path to hostfxr_initialize_parameters.dotnet_root. Also I used this in app.exe.runtimeconfig.json

{
  "runtimeOptions": {
    "tfm": "net5",
    "frameworks": [
      {
        "name": "Microsoft.NETCore.App",
        "version": "5.0.0"
      },
      {
        "name": "Microsoft.WindowsDesktop.App",
        "version": "5.0.0"
      }
    ],
    "configProperties": {
      "System.GC.Concurrent": true
    }
  }
}```
I also renamed 5.0.0-preview to jut 5.0.0 in Runtime directory. 
Is it correct approach?

[vitek-karas]

1. In general yes, but it won't work for self-contained (see Get core-setup building in the consolidated repo. #2 below) - also I personally would not use CreateInstanceFrom - it calls Assembly.LoadFrom which has "old" .NET Framework semantics and sometimes doesn't play nice with new .NET Core code (for example if you ever had assembly which has dependency on some cross-platform NuGet package this will run into trouble). I would be more explicit and directly call AssemblyLoadContext.Default.LoadFromAssemblyPath and the handle dependency resolution on my own in AssemblyLoadContext.Default.Resolving event handler - this gives you full control and avoids any surprises.
   Also this loads the code into secondary ALC - so you will have a bit of a confusion as to what runs where - your "loader" code will run in secondary ALC, but then your loaded code will run default ALC because Assembly.LoadFrom (and thus CreateInstanceFrom) always loads to Default ALC.
   I must admit I don't understand why you have a split into two assemblies - why Create loads a second assembly which does the actual work - but it more or less doesn't matter for this discussion.

2. If this is your solution to "Self-contained" then this is not right. For one the fact that your runtimeconfig has "frameworks" in it automatically means that it is framework dependent and the host will load the framework from a shared location (Program Files by default) - the fact that you copy all the assemblies locally has next to no effect on that (it might actually cause confusion and some weird behavior). You can check this by looking at where it loaded coreclr.dll from for example.

Changing 5.0.0-preview to 5.0.0 is perfectly fine - final .NET 5 SDK will generate 5.0.0 anyway.

In your solution I assume you called hostfxr_initialize_for_runtime_config currently (even in .NET 5) this function will not let you load self-contained app/component - it will fail on it. This is intentional, as this API is for loading components and self-contained components are not a solved or supported problem in .NET Core (yet). So the only supported way to load self-contained app is via hostfxr_initialize_for_dotnet_command_line. On top of that, the only supported way to build a self-contained deployment is to build and application (.exe) - SDK currently sort of works if you ask it to build a self-contained classlib, but this is not supported and may have bugs in it - I would not recommend doing that.

So I would more or less keep the code on the managed side you have - I would just build it as an application (.exe) with an empty main (current limitation of being able to only do self-contained apps, not components). Then I would publish this as self-contained dotnet publish -r win-x64 --self-contained true. Then load it via hostfxr_initialize_for_runtime_config and use hdt_get_function_pointer to get to your Create method. After that the rest is the same. Note that with this you don't need to copy any files around, you don't need to edit your custom .runtimeconfig.json and so on.

[cheverdyukv]

I would like to briefly explain structure of our application.

We have main native application and many different modules. Some of these modules are native, some are written in .NET Framework. All communications between modules done via COM interfaces (even it is not 100% COM as there is no interfaces and co-classes registrations etc). So every module has factory function that returns interface to it and further communications done via this interface.

Application does not load all modules at the same time, because it will take a lot of time and most of the customers does not need even 10% of modules. So if customer would like for example to import or export file, then that module will be loaded (if it was not loaded before) and action is executed via that root interface (or one of interfaces that root interface returns).

1. Idea here is simple. I do load Loader in secondary ALC and then load DotNetTools in default ALC. After that I only use DotNetTools to load modules. And because DotNetTools loaded to default ALC I thought I can use CreateInstanceFrom safely. Unfortunately DotNetTools has some other shared dependencies and loading it in in secondary ALC creates problems later. But anyway I will change code from CreateInstanceFrom to AssemblyLoadContext.Default.LoadFromAssemblyPath as you recommended.
2. I did check and it loads runtime from correct path. Yes I'm using hostfxr_initialize_for_runtime_config and specified dotnet_root field in hostfxr_initialize_parameters. I assumed it is what this parameter for. And our main executable is native app.
3. Are you suggesting to build Loader using dotnet publish -r win-x64 --self-contained true?
   What in this case should I use in app.exe.runtimeconfig.json and how should I specify location of runtime files to hostfxr? I really don't want to dump all 279 files in root directory of our application and prefer to keep in own directory if this is possible.
4. Just in case it is not self contained app. It is app with modules/components written in .NET. And we would like to have .NET with us to avoid problems with installing .NET as sometimes it is really pain with some enterprise customers as they assumed that if our installer displayed error about installing .NET then it is our fault and then we have to solve it and not their IT department. This
   is one of main advantages of .NET 5 over .NET Framework.

Thank you again for helping me with this!

[vitek-karas]

1. I see - I guess this should work - no complains if you need it.
2. Sorry I missed the dotnet_root setting - well, if you give it the same structure as in program files\dotnet then this is a different way to effectively achieve self-contained without building self-contained. This is definitely doable - I was kind of always hoping somebody would try this in a real-world app (you're the first I know of to try this approach). The downside of this approach is that you have to "maintain" that folder - as in copy the right files into it. But - it should be possible to simply download the "zip" version of the runtime installer and just unzip it there - that should work just fine. In any case SDK will not help you here. I haven't played with this enough - I would probably recommend that you set env. variable DOTNET_MULTILEVEL_LOOKUP=0 in your process before calling hostfxr just to make sure it doesn't try to look elsewhere (I don't think it should,... but still).
3. Yes - that was my suggestion - but with the approach in 2 this is not necessary (in fact it's counter productive) - so either 2 or this, not both. Note that the 279 files would need to be next to the loader (which you can put into any subfolder you choose), where the native app lives has no meaning here. In this case SDK would generate the .runtimeconfig.json for you - and it would be more or less empty (not really, but it would not have framework references). Note that you should not need to produce .runtimeconfig.json by hand in either case (2 or 3) - you can set EnableDynamicLoading in a classlib project and SDK will produce runtimeconfig.json for it as well.
4. I absolutely understand the appeal of self-contained. For applications it should work great out of the box - for dynamically loaded components it's a much harder problem though.

[cheverdyukv]

Thank you again for your help!

Should I to create new issue for APPDOMAIN_IGNORE_UNHANDLED_EXCEPTIONS for hostfxr or should I wait? This is my last problem with hostfxr.

[vitek-karas]

New issue might be better.

[vitek-karas]

I think this issue/question has been answered. The one remaining question is tracked separately in #39587.