MsBuild, Native binaries and AppContext.BaseDirectory.

[dazinator]

I am working on the GitVersion msbuild task. It's distrubuted as a nuget package. The nuget package contains a build of the task assembly for net461 and a build for netstandard1.5.
The corresponding libgit2sharp assembly (which our task depends on) and the /lib folder containing the native binaries that libgit2sharp requires, lives alongside the task assembly, in the nuget package.

When the nuget package is added to a project, and the project is built within VS - this uses the full desktop version of msbuild, and subsequently the net461 assembly containing the GitVersion build task is loaded from the nuget package, and the task runs successfully - LibGit2sharp all works smoothly.

However when dropping to the command line and running dotnet build (I am using version 2.0.2 of the dotnet sdk) this runs msbuild on netcoreapp and thus the netstandard1.5 task assembly from the nuget package is loaded. In this scenario the task msbuild fails and this is the stack trace:

       Using "GetVersion" task from the task factory "NuGetTaskRunnerFactory".
       Task "GetVersion"
         Resolved GitVersionTask, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null located in C:\Users\Darrell\.nuget\packages\gitversiontask\0.0.1-alpha0003\build\netstandard1.5\GitVersionTask.dll and loaded from C:\Users\Darrell\.nuget\packages\gitversiontask\0.0.1-alpha0003\build\netstandard1.5\GitVersionTask.dll.
         Resolved GitVersionCore, Version=4.0.0.0, Culture=neutral, PublicKeyToken=null located in C:\Users\Darrell\.nuget\packages\gitversiontask\0.0.1-alpha0003\build\netstandard1.5\GitVersionCore.dll and loaded from C:\Users\Darrell\.nuget\packages\gitversiontask\0.0.1-alpha0003\build\netstandard1.5\GitVersionCore.dll.
         Resolved GitTools.Core, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null located in C:\Users\Darrell\.nuget\packages\gitversiontask\0.0.1-alpha0003\build\netstandard1.5\GitTools.Core.dll and loaded from C:\Users\Darrell\.nuget\packages\gitversiontask\0.0.1-alpha0003\build\netstandard1.5\GitTools.Core.dll.
         Resolved LibGit2Sharp, Version=0.25.0.0, Culture=neutral, PublicKeyToken=7cbde695407f0333 located in C:\Users\Darrell\.nuget\packages\gitversiontask\0.0.1-alpha0003\build\netstandard1.5\LibGit2Sharp.dll and loaded from C:\Users\Darrell\.nuget\packages\gitversiontask\0.0.1-alpha0003\build\netstandard1.5\LibGit2Sharp.dll.
     1>MSBUILD : warning : WARN [10/24/17 20:59:04:58] Could not determine assembly version: System.TypeInitializationException: The type initializer for 'LibGit2Sharp.Core.NativeMethods' threw an exception. ---> System.DllNotFoundException: Unable to load DLL 'git2-15e1193': The specified module could not be found. (Exception from HRESULT: 0x8007007E)\r [C:\Users\Darrell\Documents\visual studio 2017\Projects\ClassLibrary7\ClassLibrary7.csproj]
MSBUILD : warning :    at LibGit2Sharp.Core.NativeMethods.git_libgit2_init()\r [C:\Users\Darrell\Documents\visual studio 2017\Projects\ClassLibrary7\ClassLibrary7.csproj]
MSBUILD : warning :    at LibGit2Sharp.Core.NativeMethods.LoadNativeLibrary()\r [C:\Users\Darrell\Documents\visual studio 2017\Projects\ClassLibrary7\ClassLibrary7.csproj]
MSBUILD : warning :    at LibGit2Sharp.Core.NativeMethods..cctor()\r [C:\Users\Darrell\Documents\visual studio 2017\Projects\ClassLibrary7\ClassLibrary7.csproj]
MSBUILD : warning :    --- End of inner exception stack trace ---\r [C:\Users\Darrell\Documents\visual studio 2017\Projects\ClassLibrary7\ClassLibrary7.csproj]
MSBUILD : warning :    at LibGit2Sharp.Core.NativeMethods.git_buf_free(GitBuf buf)\r [C:\Users\Darrell\Documents\visual studio 2017\Projects\ClassLibrary7\ClassLibrary7.csproj]
MSBUILD : warning :    at LibGit2Sharp.Core.Proxy.ConvertPath(Func`2 pathRetriever)\r [C:\Users\Darrell\Documents\visual studio 2017\Projects\ClassLibrary7\ClassLibrary7.csproj]
MSBUILD : warning :    at LibGit2Sharp.Repository.Discover(String startingPath)\r [C:\Users\Darrell\Documents\visual studio 2017\Projects\ClassLibrary7\ClassLibrary7.csproj]
MSBUILD : warning :    at GitVersion.GitPreparer.GetDotGitDirectory() in C:\Users\Darrell\Source\Repos\GitVersion\src\GitVersionCore\GitPreparer.cs:line 139\r [C:\Users\Darrell\Documents\visual studio 2017\Projects\ClassLibrary7\ClassLibrary7.csproj]
MSBUILD : warning :    at GitVersion.ExecuteCore.ExecuteGitVersion(String targetUrl, String dynamicRepositoryLocation, Authentication authentication, String targetBranch, Boolean noFetch, String workingDirectory, String commitId, Config overrideConfig, Boolean noCache) in C:\Users\Darrell\Source\Repos\GitVersion\src\GitVersionCore\ExecuteCore.cs:line 32\r [C:\Users\Darrell\Documents\visual studio 2017\Projects\ClassLibrary7\ClassLibrary7.csproj]
MSBUILD : warning :    at GitVersion.ExecuteCore.TryGetVersion(String directory, VersionVariables& versionVariables, Boolean noFetch, Authentication authentication) in C:\Users\Darrell\Source\Repos\GitVersion\src\GitVersionCore\ExecuteCore.cs:line 81 [C:\Users\Darrell\Documents\visual studio 2017\Projects\ClassLibrary7\ClassLibrary7.csproj]
       Done executing task "GetVersion".

From the stack trace, it shows that the libgit2sharp assembly is being loaded from alongside the task assembly: C:\Users\Darrell\.nuget\packages\gitversiontask\0.0.1-alpha0003\build\netstandard1.5\LibGit2Sharp.dll

It also shows this:

System.TypeInitializationException: The type initializer for 'LibGit2Sharp.Core.NativeMethods' threw an exception. ---> System.DllNotFoundException: Unable to load DLL 'git2-15e1193': The specified module could not be found. (Exception from HRESULT: 0x8007007E)\r [C:\Users\Darrell\Documents\visual studio 2017\Projects\ClassLibrary7\ClassLibrary7.csproj]
MSBUILD : warning :    at LibGit2Sharp.Core.NativeMethods.git_libgit2_init()\r [C:\Users\Darrell\Documents\visual studio 2017\Projects\ClassLibrary7\ClassLibrary7.csproj]
MSBUILD : warning :    at LibGit2Sharp.Core.NativeMethods.LoadNativeLibrary()\r 

Yet alongside the libgit2sharp assembly I have confirmed that there is a lib folder containing all of the native binaries - including git2-15e1193.dll:

Any ideas? @bording ?

If you need to replicate, you can do so by taking my PR branch of gitversion, calling msbuild /t:Pack on the msbuild task project to produce a nuget package containing libgit2sharp, then adding that nuget package to a sample .csproj file that has an initialised .git repository with atleast 1 commit, and building.

[dazinator]

This looks like the culprit:

libgit2sharp/LibGit2Sharp/GlobalSettings.cs

Line 49 in 72fa368

string managedPath = AppContext.BaseDirectory;

When running under .netcoreapp, libgit2sharp uses AppContext.BaseDirectory - which is going to be pointing to the msbuild.exe directory no doubt.. This would only work for top level applications, not for nuget packages.

[dazinator]

If someone agrees that this is the problem, I don't mind taking a stab at a PR to fix this.

[ethomson]

Interesting! Good detective work! For sure if you have a proposed fix I'd be very interested! 😀

[bording]

@dazinator To be honest, the current state of the repo is a bit fragile because of the code generation stuff going on to get it to work on netstandard < 2.0. That's one of the reasons I've opened #1483.

The intention when using this from .NET Core is that the native dependencies that are shipped in the native binaries package are directly referenced using the native assets capabilities directly. The fact that you see the lib folder showing up is an artifact of the current prerelease package being released before libgit2/libgit2sharp.nativebinaries#49 got merged.

@ethomson We really do need to get an updated package out there that includes those changes.

So, @dazinator, if you take a look at a netcoreapp2.0 project that references LibGit2Sharp and then publish it, you'll see the runtimes folder in there. That's where the native loading capabilities need the files to work correctly.

[dazinator]

In terms of a solution it looks.. tricky.

This is relevent - i.e Assembly.Location not being available in netstandard1.3: https://github.com/dotnet/corefx/issues/8398

It was mentioned above that you can still get hold of Assembly.Location via reflection even though it's not part of the netstandard api - as I beleive its only UWP that doesn't support it.

So one option would be to change the compilation directive to something like this:

 if (Platform.OperatingSystem == OperatingSystemType.Windows)
            {
               
                string managedPath = string.Empty;

#if DESKTOP
                var assembly = Assembly.GetExecutingAssembly();
                managedPath = assembly.CodeBase;
                if (managedPath == null)
                {
                    managedPath = assembly.Location;
                }
#else
                var assembly = typeof(GlobalSettings).GetTypeInfo().Assembly;
                managedPath = (string)assembly.GetType().GetProperty("Location").GetValue(assembly);              
#endif


                /* Assembly.CodeBase is not actually a correctly formatted
                 * URI.  It's merely prefixed with `file:///` and has its
                 * backslashes flipped.  This is superior to EscapedCodeBase,
                 * which does not correctly escape things, and ambiguates a
                 * space (%20) with a literal `%20` in the path.  Sigh.
                 */

                if (managedPath.StartsWith("file:///"))
                {
                    managedPath = managedPath.Substring(8).Replace('/', '\\');
                }
                else if (managedPath.StartsWith("file://"))
                {
                    managedPath = @"\\" + managedPath.Substring(7).Replace('/', '\\');
                }

                managedPath = Path.GetDirectoryName(managedPath);
                nativeLibraryPath = Path.Combine(managedPath, "lib", "win32");
            }

            registeredFilters = new Dictionary<Filter, FilterRegistration>();
        }

Note: this requires adding a new dependency for netstandard1.3:

 <PackageReference Include="System.Reflection.TypeExtensions" Version="4.4.0" />

I have just tried this, and all the unit tests still pass after this change.

Another option would be to add a target for netstandard1.5 where Assembly.Location is available. Then atleast anyone in my position is able to avoid the problem as long as they are on netstandard level 1.5 or higher.

[bording]

In other words, the lib folder is intentionally not relevant for .NET Core, and the next release of the package will have it stop needlessly copying files into the lib folder for those cases.

[dazinator]

@bording - sorry I think we started posting at the same time.

Would you please enlighten me - if the /lib folder is irrelevent in a future release (for netcoreapp), can you explain how libgit2sharp will locate the native libs in my scenario. Will they be embedded into the libgit2sharp assembly or something instead? Or is there some other magical mechanism! 😄

[bording]

@dazinator It's relying on .NET Core's built-in understanding of how to find and load native dependencies that are shipped in NuGet packages:

The appropriate native library is referenced based on the RID. If you dotnet run or F5 in VS, then those are loaded directly from the copy in the global nuget package cache.

If you publish first, then you'll see the runtimes folder copied in the publish output. That combined with the deps.json file is enough for .NET Core to find and and load the correct file, once again based on the RID.

[bording]

I've got another open PR libgit2/libgit2sharp.nativebinaries#51 to further leverage this capability to be able to ship multiple files for different linux distros as well.

[dazinator]

@bording

It's relying on .NET Core's built-in understanding of how to find and load native dependencies that are shipped in NuGet packages:

My concern is that in this case msbuild is loading the task assembly from an arbitrary location (a path in a targets file points at an assembly on disk that happens to be delivered via a nuget package). My concern is that when msbuild loads the task from the assembly on disk, it doesn't look for a corresponding deps file so it won't hook this up to netcore's resolution logic that you are talking about..

Does the deps file need to be included in the nuget package that we ship the msbuild task in?

[bording]

@dazinator It's my understanding that this should work for anything running in .NET Core, including the .NET Core version of MSBuild, but it's always possible there's some scenario there that isn't properly handled.

[bording]

These sort of loading complexities are what I had in mind when I was suggesting moving to a DotnetCliTool in GitTools/GitVersion#1269 (comment)

[dazinator]

It's my understanding that this should work for anything running in .NET Core, including the .NET Core version of MSBuild, but it's always possible there's some scenario there that isn't properly handled.

I would love it to just work.. But I retain a healthy skepticism :-)
It would need to work with all msbuild versions on netcoreapp (netcoreapp1.0, 1.1, 2.0 etc) because we can;t guarantee which version of the dotnet sdk people will use to build their projects with. It could be anything from 1.0.0 to 2.0.2 and beyond.

I had previously packaged just my task assembly (for netstandard1.5) and not its depedencies in the nuget package, and there was a deps file too. When doing a dotnet build, Msbuild wasn't clever enough to resolve my task assemblies dependencies, so it made me believe that it doesn't know how to resolve dependencies under netcore full stop irrespective of any deps files that happen to live alongside the task assembly. However I could be doing something wrong - this whole thing has been an absolute nightmare.

These sort of loading complexities are what I had in mind when I was suggesting moving to a DotnetCliTool in GitTools/GitVersion#1269 (comment)

Yep - it's a good suggestion - I just wasn't sure how to best go about bringing it into reality hence my probing comments! You are welcome to elaborate / contribute / spike up something over on gitversion repo.

I will be happy to try out this new loading logic from an msbuild task as soon as its available please let me know!

Please also consider my suggestions above in terms of tweaks to the code to potentially unblock us. I am happy to include the lib folder alongside the libgit2sharp assembly in the nuget package - the tweak would just allow it to be discovered similar to how netdesktop behaves. I am happy to submit a PR for this.

[dazinator]

PR created. I verified the fix by including the modified libgit2sharp assembly in my nuget package, with my netstandard1.5 msbuild task assembly, and the /lib folder. My build task then ran fine under netcore and desktop versions of msbuild as libgit2sharp was now able to locate the /lib.

[dazinator]

I have closed my PR as it resolved it only on windows. Need a solution that works on all supported platforms.

[bording]

I'm going to go ahead and close this since using the lib folder and altering the path is not something that we'll be changing for .NET Core.