Build failing when Adding Npgsql

[lpeixotoo]

Hello,

I keep getting the same problem when running the HelloWorld Sample with Npgsql.

Running dotnet publish -r linux-x64 -c Release -v d

I get this error during the "Exec" task of the publish command:
Task "Exec" "/root/.nuget/packages/runtime.linux-x64.microsoft.dotnet.ilcompiler/1.0.0-alpha-27906-01/tools/ilc" @"obj/Release/netcoreapp2.1/linux-x64/native/DbPrototype.ilc.rsp" Killed 1:7>/root/.nuget/packages/microsoft.dotnet.ilcompiler/1.0.0-alpha-27906-01/build/Microsoft.NETCore.Native.targets(248,5): error MSB3073: The command ""/root/.nuget/packages/runtime.linux-x64.microsoft.dotnet.ilcompiler/1.0.0-alpha-27906-01/tools/ilc" @"obj/Release/netcoreapp2.1/linux-x64/native/DbPrototype.ilc.rsp"" exited with code 137. [/DbPrototype/DbPrototype.fsproj] Done executing task "Exec" -- FAILED. 1:7>Done building target "IlcCompile" in project "DbPrototype.fsproj" -- FAILED. 1:7>Done Building Project "/DbPrototype/DbPrototype.fsproj" (Publish target(s)) -- FAILED.

OS: Ubuntu 18.04 (All pre requisites installed based on Helloworld sample)

[jkotas]

What is the Npgsql version that you are using? Would you mind to share the code and .csproj file that reproduces the issue?

This crash looks like the infinite generic recursion that was fixed in more recent Npgsql versions: #6372 (comment) .

[lpeixotoo]

Sorry I forgot to list those here.

Here's my .fsproj file

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp2.1</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <Compile Include="Program.fs" />
  </ItemGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.DotNet.ILCompiler" Version="1.0.0-alpha-*" />
    <PackageReference Include="Npgsql" Version="4.0.7" />
    <PackageReference Include="Npgsql.Fsharp" Version="1.10.0" /> 
  </ItemGroup>

</Project>

My Program.fs is just the Helloworld one, here it is:

open System

[<EntryPoint>]
let main argv =
    printfn "Hello World from F#!"
    0 // return an integer exit code

[jkotas]

It does not crash for me. Here is what I have done to try to reproduce this:

docker run -it mcr.microsoft.com/dotnet/core/sdk:2.1

apt-get update
apt-get install clang-3.9 libcurl4-openssl-dev zlib1g-dev libkrb5-dev

dotnet new console -lang F# -o hello
cd hello

<replace hello.fsproj with above>
<add https://github.com/dotnet/corert/blob/master/samples/HelloWorld/nuget.config>

dotnet publish -c Release -r linux-x64

What is the Linux distro that you are seeing the crash on?

[lpeixotoo]

Hi @jkotas ,

I'm sorry for my disorganisation. Here's a repo that I created regarding this issue. It actually uses Ubuntu 18.04 base image but I found the same issue using .Net Core image(Ubuntu).

[jkotas]

You are missing libkrb5-dev in the list of dependencies. Once I have added it to your docker file, it built successfully for me.

diff --git a/Dockerfile b/Dockerfile
index 31b310d..5f9604e 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -35,7 +35,8 @@ RUN apt-get install -y \
         #tk-dev \
         #wget \
         #xz-utils \
-        zlib1g-dev
+        zlib1g-dev \
+        libkrb5-dev

 WORKDIR /home/app

[lpeixotoo]

So I've added the library and built successfully for me too. Then I tried to connect into the DB and execute a simple SQL command and for that, it crashes again. See this commit. I've also added docker-compose file so It could be easier to build too.

[jkotas]

What is the crash that you are see with the latest change?

[lpeixotoo]

It's the same one that we were having at the first place

Task "Exec"
         "/root/.nuget/packages/runtime.linux-x64.microsoft.dotnet.ilcompiler/1.0.0-alpha-27910-01/tools/ilc" @"obj/Release/netcoreapp2.1/linux-x64/native/DbPrototype.ilc.rsp"
		         Killed
   1:7>/root/.nuget/packages/microsoft.dotnet.ilcompiler/1.0.0-alpha-27910-01/build/Microsoft.NETCore.Native.targets(249,5): error MSB3073: The command ""/root/.nuget/packages/runtime.linux-x64.microsoft.dotnet.ilcompiler/1.0.0-alpha-27910-01/tools/ilc" @"obj/Release/netcoreapp2.1/linux-x64/native/DbPrototype.ilc.rsp"" exited with code 137. [/home/app/DbPrototype.fsproj]
       Done executing task "Exec" -- FAILED.
   1:7>Done building target "IlcCompile" in project "DbPrototype.fsproj" -- FAILED.
   1:7>Done Building Project "/home/app/DbPrototype.fsproj" (Publish target(s)) -- FAILED.

[jkotas]

I am not able to reproduce this crash. Does it deterministically crash for you, or is the crash intermittent?

[lpeixotoo]

Intermittently. Were you able to compile it using the last commit too? Via docker compose? I'm already using docker into a OSX system.

[jkotas]

I was able to compile using docker compose. It compiled fine for me.

The error I got was this one:

database_1   | running bootstrap script ... ok
prototype_1  | Unhandled Exception: System.TypeInitializationException: A type initializer threw an exception. To determine which type, inspect the InnerException's StackTrace property.
prototype_1  |  ---> EETypeRva:0x05B6B9C8(System.Reflection.MissingRuntimeArtifactException): MakeGenericMethod() cannot create this generic method instantiation because the instantiation was not metadata-enabled: 'Microsoft.FSharp.Core.PrintfImpl.Specializations<Microsoft.FSharp.Core.Unit,System.String,System.String>.FinalFastEnd5<System.String,System.String,System.String,System.Int32,System.String>(System.String,Microsoft.FSharp.Core.FSharpFunc<System.String,System.String>,System.String,Microsoft.FSharp.Core.FSharpFunc<System.String,System.String>,System.String,Microsoft.FSharp.Core.FSharpFunc<System.String,System.String>,System.String,Microsoft.FSharp.Core.FSharpFunc<System.Int32,System.String>,System.String,Microsoft.FSharp.Core.FSharpFunc<System.String,System.String>)' For more information, please visit http://go.microsoft.com/fwlink/?LinkID=616868
prototype_1  |    at Internal.Reflection.Core.Execution.ExecutionEnvironment.GetMethodInvoker(RuntimeTypeInfo, QMethodDefinition, RuntimeTypeInfo[], MemberInfo) + 0x165
prototype_1  |    at System.Reflection.Runtime.MethodInfos.NativeFormat.NativeFormatMethodCommon.GetUncachedMethodInvoker(RuntimeTypeInfo[], MemberInfo) + 0x3f
prototype_1  |    at System.Reflection.Runtime.MethodInfos.RuntimeMethodInfo.get_MethodInvoker() + 0xab
prototype_1  |    at System.Reflection.Runtime.MethodInfos.RuntimeNamedMethodInfo`1.MakeGenericMethod(Type[]) + 0x114
prototype_1  |    at Microsoft.FSharp.Core.PrintfImpl.PrintfBuilder`3.buildPlainFinal(Object[], Type[]) + 0x243
prototype_1  |    at Microsoft.FSharp.Core.PrintfImpl.PrintfBuilder`3.parseFromFormatSpecifier(String, String, Type, Int32) + 0x517
prototype_1  |    at Microsoft.FSharp.Core.PrintfImpl.PrintfBuilder`3.parseFromFormatSpecifier(String, String, Type, Int32) + 0x1ed
prototype_1  |    at Microsoft.FSharp.Core.PrintfImpl.PrintfBuilder`3.parseFormatString(String, Type) + 0xb4
prototype_1  |    at Microsoft.FSharp.Core.PrintfImpl.PrintfBuilder`3.Build[T](String) + 0x42
prototype_1  |    at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey, Func`2) + 0x80
prototype_1  |    at Microsoft.FSharp.Core.PrintfImpl.Cache`4.Get(PrintfFormat`4) + 0x6b
prototype_1  |    at Microsoft.FSharp.Core.PrintfModule.PrintFormatToStringThen[T](PrintfFormat`4) + 0x19
prototype_1  |    at Npgsql.FSharp.SqlModule.str(SqlModule.ConnectionStringBuilder) + 0x3d
prototype_1  |    at <StartupCode$DbPrototype>.$Program..cctor() + 0x1ca
prototype_1  |    at System.Runtime.CompilerServices.ClassConstructorRunner.EnsureClassConstructorRun(StaticClassConstructionContext*) + 0xd5
prototype_1  |    --- End of inner exception stack trace ---
prototype_1  |    at System.Runtime.CompilerServices.ClassConstructorRunner.EnsureClassConstructorRun(StaticClassConstructionContext*) + 0x198
prototype_1  |    at System.Runtime.CompilerServices.ClassConstructorRunner.CheckStaticClassConstructionReturnNonGCStaticBase(StaticClassConstructionContext*, IntPtr) + 0x9
prototype_1  |    at Program.main(String[]) + 0xd
prototype_1  |    at DbPrototype!<BaseAddress>+0x1896b6d
prototype_1  | Aborted
database_1   | performing post-bootstrap initialization ... ok

This is a known problem caused by extensive use of reflection on generics inside FSharp implementation of PrintfF that is not friendly to AOT compilation. Discussed in #2057 .

[lpeixotoo]

Hi @jkotas,

So I refactored the code in the issue repo with a C# and F# version.

• The C# version I was able to compile but It generates an exception at runtime. I think it's related
  to type reflections. Do we have a solution for that using rd.xml? If yes, do we any directive on how to do it?

• The F# version I wasn't able to compile it (Infinite generic recursion growing into the system's memory), tried it on both OSX and Linux. This is the same that you've compiled. Have you used any config within docker to be able to compile it? If so, could list them so I can compile it too?

[jkotas]

Do we have a solution for that using rd.xml?

Some progress on .rd.xml required to make Npgsql work was made in #6784 , but it was not fully solved.

Have you used any config within docker to be able to compile it?

I have not done anything special. Just run docker-compose up.

[lpeixotoo]

I was able to compile it too! I've to bump up the docker environment memory to 6GB. But for now I'm getting the same exception from #6784, @MichalStrehovsky Is this the stack trace that we are supposed to look at?

Please regard to this branch of the F# version to reproduce it, if you want to.

prototype_1  | Unhandled Exception: System.TypeInitializationException: A type initializer threw an exception. To determine which type, inspect the InnerException's StackTrace property.
prototype_1  |  ---> System.TypeInitializationException: A type initializer threw an exception. To determine which type, inspect the InnerException's StackTrace property.
prototype_1  |  ---> System.NullReferenceException: Object reference not set to an instance of an object.
prototype_1  |    at Npgsql.Logging.NpgsqlLogManager.GetClassFullName() + 0x54
prototype_1  |    at Npgsql.NpgsqlConnection..cctor() + 0x57
prototype_1  |    at System.Runtime.CompilerServices.ClassConstructorRunner.EnsureClassConstructorRun(StaticClassConstructionContext*) + 0xd1
prototype_1  |    --- End of inner exception stack trace ---
prototype_1  |    at System.Runtime.CompilerServices.ClassConstructorRunner.EnsureClassConstructorRun(StaticClassConstructionContext*) + 0x194
prototype_1  |    at System.Runtime.CompilerServices.ClassConstructorRunner.CheckStaticClassConstructionReturnGCStaticBase(StaticClassConstructionContext*, Object) + 0x9
prototype_1  |    at Npgsql.NpgsqlConnection..ctor() + 0x29
prototype_1  |    at <StartupCode$HelloWorld>.$Program..cctor() + 0x20
prototype_1  |    at System.Runtime.CompilerServices.ClassConstructorRunner.EnsureClassConstructorRun(StaticClassConstructionContext*) + 0xd1
prototype_1  |    --- End of inner exception stack trace ---
prototype_1  |    at System.Runtime.CompilerServices.ClassConstructorRunner.EnsureClassConstructorRun(StaticClassConstructionContext*) + 0x194
prototype_1  |    at System.Runtime.CompilerServices.ClassConstructorRunner.CheckStaticClassConstructionReturnNonGCStaticBase(StaticClassConstructionContext*, IntPtr) + 0x9
prototype_1  |    at HelloWorld!<BaseAddress>+0x194a00a

[MichalStrehovsky]

@OshoNot Try adding <IlcArg Include="--nometadatablocking" /> to an ItemGroup in your project file.

Npgsql is essentially reflecting on the managed class constructor runner in CoreRT (an implementation detail of CoreRT that is not visible to reflection unless a switch is passed to the compiler). The class constructor runner in CoreCLR is in native code and stack trace doesn't show it. Stack frames that contain CoreRT implementation details are a bit of a pain because they are not useful anyway. I advocated just eliminating them so that we're more like CoreCLR in this matter, but instead we built a whole new feature that makes ToString() for these stack frames work, but they still fail miserably for GetMethod(). It's unfortunate.

[roji]

@MichalStrehovsky thanks for following up on this and for the explanations.

What would you suggest we do here? This stack traversal code is non-essential - it's just a way to set up loggers with their class names - so I could just remove it. On the other hand, it does seem like something that should be fixed in CoreRT.

[roji]

Actually, thinking about this again, it seems like smelly practice inherited from the times before nameof() existed - I'll remove it regardless of CoreRT support.

[MichalStrehovsky]

Actually, thinking about this again, it seems like smelly practice inherited from the times before nameof() existed - I'll remove it regardless of CoreRT support.

Awesome. It looks like a pretty fragile piece of code anyway - more aggressive inlining would change what the logger reports.

Otherwise, the fix would be to check the result of GetMethod() for null. The method is annotated as returning null - so not checking for it would be a compile failure with nullable reference types turned on anyway:

https://github.com/dotnet/coreclr/blob/5f93d3b1c48ba6916d5f31d79cb7c17d564eecef/src/System.Private.CoreLib/shared/System/Diagnostics/StackFrame.cs#L129

[roji]

It looks like a pretty fragile piece of code anyway

Absolutely, this is quite ancient stuff.

Is there any sense in keeping this open, to track fully supporting stack frame reflection in CoreRT? Also, can we also close #6784 as a dup of this (the F#-specific issues seem to be already tracked by #2057).

[lpeixotoo]

@MichalStrehovsky,
I tried to add it and results in the same error.
@roji,
I'll keep track of npgsql/npgsql#2542. Could you guys put it on a unstable feed after removing? So I can reproduce it and update the issue related branch repo.

[MichalStrehovsky]

to track fully supporting stack frame reflection in CoreRT

It is fully supported in a way. The "problem" with CoreRT is that most of the runtime is written in managed code (which is really a good thing) and the managed pieces become visible in places like this.

Things that are in native code in CoreCLR get shielded from reflection and code like the one in Npgsql works.

To really fix this, we would have to annotate managed code in CoreRT as "things that are in native code on CoreCLR" so that we can properly emulate what stack trace does.

The --nometadatablocking that I suggested might make the code not crash with a null reference, but it still won't do the right thing because on CoreCLR, it wouldn't report the class constructor runner frame (ClassConstructorRunner.EnsureClassConstructorRun).

I don't think we would ever want to go in that direction.

[roji]

@MichalStrehovsky OK, sounds like this and #6784 should be closed - will let you do the honors.