Microsoft.AspNetCore.Hosting throws Startup assembly StartupBootstrapper failed to execute

[ghost]

I see these errors logged on Azure (App Service) on every restart of the App Service:

Main message:

Startup assembly StartupBootstrapper failed to execute. See the inner exception for more details. Could not load file or assembly 'StartupBootstrapper, Culture=neutral, PublicKeyToken=null'. The system cannot find the file specified. 

Stack trace:

System.InvalidOperationException:
System.IO.FileNotFoundException:
   at System.Reflection.RuntimeAssembly.nLoad (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Reflection.Assembly.Load (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Reflection.Assembly.Load (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.ExecuteHostingStartups (Microsoft.AspNetCore.Hosting, Version=3.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)

Let me know what additional data I can collect for you that may help in fixing this issue.

[davidfowl]

This happens when the IHostingStartup in site extensions fail to load. StartupBootstrapper is an application insights type see https://github.com/Microsoft/ApplicationInsights-Home/issues/231.

[davidfowl]

BTW is your application self contained?

[ghost]

@davidfowl :

I tried reading https://github.com/microsoft/ApplicationInsights-Home/issues/231 and the attached issue #5919. It is unclear what exactly resolved the issue -- and if in fact the issue was resolved in those cases.

Yes, my application is self-contained, here's the config:

Done (and desired) by me, manually:

• Azure App Service, Windows environment, Code deployment
• Code built on .NET Core 3.0 Preview, C# 8.0 Preview (cutting bleeding edge!) using Azure DevOps CI pipeline from a GitHub repo.
• Application Insights installed into app via Visual Studio's Connected Services > Application Insights page. It shows "100% configured".
• Application Insights is enabled in "full" mode (everything except SQL collection is ON) in Azure Portal.

I didn't want to go into Azure Portal and keep updating extensions there, so I haven't installed the .NET Core 3.0 Site Extension -- this is why I have it deployed in SCD mode.

[davidfowl]

Unfortunately right now, the site extensions do not work with self contained deployments. You'll need to add the Application Insights SDK manually to get it working.

I didn't want to go into Azure Portal and keep updating extensions there, so I haven't installed the .NET Core 3.0 Site Extension -- this is why I have it deployed in SCD mode.

I think you'd need to remove the site extension to prevent it from throwing at startup.

[ghost]

The only Site Extension present on this app service (as per App Service > Extensions blade on the portal is ASP.NET Core Logging Extensions of version 2.2.0. And, I have not installed this myself, it is something Azure does.

Could you clarify what you mean by "You'll need to add the Application Insights SDK manually" ? I have installed it from within the Visual Studio IDE.

[davidfowl]

OK I spent some time looking into this and you can't get rid of this exception today if the application is self contained. The feature being used to inject application insights into the application always fails on startup (because of a limitation in the current design).

The only way to get rid of it today is to do a framework dependent deployment by either waiting for 3.0 to be installed on app service or by using the ASP.NET Core runtime site extension.

I'll keep this issue open and assign it to myself as it's something we need to solve.

[ghost]

Thanks @davidfowl.

Strange thing is, the Application Insights seems to be working fine. I see the traffic, exceptions and everything else as I normally should. This means App Insights is still being loaded?

[davidfowl]

Strange thing is, the Application Insights seems to be working fine. I see the traffic, exceptions and everything else as I normally should. This means App Insights is still being loaded?

Yes, it works fine but it'll still try to load some other pieces of code at startup and always fail.

[ghost]

Tangential question - Is there a way to simulate how an self-contained deployment would work in a local (laptop/PC) environment?

[davidfowl]

What do you mean exactly? You can self contain publish and then run the published application locally? Is there something more you were looking for?

[ghost]

Let's say you need to troubleshoot a problem (like the topic of this assembly load). You'd need an environment that mimics it right? So how would you mimic an SCD environment?

When I publish and run an SCD application locally, the runtime is already present globally because of my primary .NET Core installation.

[davidfowl]

Its doesnt matter if the runtime is already present globally it won't be used. The application produced has no ties to the globally installed runtime.

[analogrelay]

I'll keep this issue open and assign it to myself as it's something we need to solve.

What's the action you want to take here @davidfowl ?

[davidfowl]

We're working with the App insights team on resolving this for 3.0.

[analogrelay]

Ok, so at the moment there is no action on our side. I'll leave this open to track that work though.

[analogrelay]

Any update or issue tracking this @davidfowl ? Can you loop me in on conversations related to this?

[dradovic]

I'm facing the same issue after upgrading our web app from 2.1 to 3.0. However, it is not self-contained. Shall I open a new issue for that case?

[analogrelay]

I'm facing the same issue after upgrading our web app from 2.1 to 3.0. However, it is not self-contained. Shall I open a new issue for that case?

This is likely part of #15381 (the issue refers to 5.0 but there's a 3.0 problem as well).

[LawrenceSmith1437]

Note, this is reproducible in Core 3.1.

[SidShetye]

We're seeing this issue on Azure App Service Windows Servers + .NET Core 3.1 + FDD (not SCD as above). The same app deployed to Azure App Service Linux Containers + .NET Core 3.1 + FDD does not have this issue. There is also another immediate log entry related to Microsoft.AspNetCore.AzureAppServices.HostingStartup, copying that here too for SEO and other folks hitting this issue.

What's the timeline to resolution? It's holding up our production migration to 3.1 since it's LTS.

------------------------------
2019-12-12 23:31:06.885 +00:00 [Critical] Microsoft.AspNetCore.Hosting.Diagnostics:
Hosting startup assembly exception
System.InvalidOperationException: Startup assembly DiagnosticServices.HostingStartup failed to execute. See the inner exception for more details.
 ---> System.IO.FileNotFoundException: Could not load file or assembly 'DiagnosticServices.HostingStartup, Culture=neutral, PublicKeyToken=null'. The system cannot find the file specified.
File name: 'DiagnosticServices.HostingStartup, Culture=neutral, PublicKeyToken=null'
   at System.Reflection.RuntimeAssembly.nLoad(AssemblyName fileName, String codeBase, RuntimeAssembly assemblyContext, StackCrawlMark& stackMark, Boolean throwOnFileNotFound, AssemblyLoadContext assemblyLoadContext)
   at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, StackCrawlMark& stackMark, AssemblyLoadContext assemblyLoadContext)
   at System.Reflection.Assembly.Load(AssemblyName assemblyRef, StackCrawlMark& stackMark, AssemblyLoadContext assemblyLoadContext)
   at System.Reflection.Assembly.Load(AssemblyName assemblyRef)
   at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.ExecuteHostingStartups()


   --- End of inner exception stack trace ---
------------------------------
2019-12-12 23:31:07.897 +00:00 [Critical] Microsoft.AspNetCore.Hosting.Diagnostics:
Hosting startup assembly exception
System.InvalidOperationException: Startup assembly Microsoft.AspNetCore.AzureAppServices.HostingStartup failed to execute. See the inner exception for more details.
 ---> System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.AspNetCore.AzureAppServices.HostingStartup, Culture=neutral, PublicKeyToken=null'. The system cannot find the file specified.
File name: 'Microsoft.AspNetCore.AzureAppServices.HostingStartup, Culture=neutral, PublicKeyToken=null'
   at System.Reflection.RuntimeAssembly.nLoad(AssemblyName fileName, String codeBase, RuntimeAssembly assemblyContext, StackCrawlMark& stackMark, Boolean throwOnFileNotFound, AssemblyLoadContext assemblyLoadContext)
   at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, StackCrawlMark& stackMark, AssemblyLoadContext assemblyLoadContext)
   at System.Reflection.Assembly.Load(AssemblyName assemblyRef, StackCrawlMark& stackMark, AssemblyLoadContext assemblyLoadContext)
   at System.Reflection.Assembly.Load(AssemblyName assemblyRef)
   at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.ExecuteHostingStartups()


   --- End of inner exception stack trace ---

[davidfowl]

What's the timeline to resolution? It's holding up our production migration to 3.1 since it's LTS.

What's holding up anything? The application should still be working. Is that not the case?

[SidShetye]

@davidfowl To be honest, the application does bootup and we see it working fine under full load test. But we're also seeing some automatic restart every a 4-6 hours, under 0% load. This kills our <200 ms response time SLAs as cold restarts are clocking in at 45 seconds (haven't looked into ready2run optimizations yet).

Back to attribution, logging shows only the following smoking guns:

1. App layer: this Application Insight issue (i.e. Microsoft.AspNetCore.Hosting throws Startup assembly StartupBootstrapper failed to execute #14547) which we're seeing only on the ASP.NET Core 3.1 + Windows OS.
2. App layer: Another Application insight issue (The logged trace number doesn't match the valid trace number microsoft/ApplicationInsights-Profiler-AspNetCore#66). This is observed on both Linux and Windows OS.
3. Host layer: On App Service for Linux (=Linux containers) we also seeing Stoping site <sitename> because it exceeded swap usage limits. at 0% load. That results in a container recycle despite the portal's 'diagnose and solve problems' wizard showing < 500 MB use on a 3.5 GB app service plan/instance. This happens with the standard Linux container (that azure builds for us when we offer it code) as well as our custom container (but less frequently). We're still trying to figure out if this issue is rooted in one of the above two AppInsight issues.

We're not seeing any other log warn/critical/fatal messages from any other sub-system and components, hence the hesitation to go to production with azure app service (linux or windows), asp.net core 3.1 and appinsights till we can root cause and eliminate the app restarts.

[analogrelay]

@SidShetye Perhaps the best thing for you to do here would be to engage Azure Support to dig deeper in to what's going on in your specific scenario. It sounds like you're hitting the same issue, but along with some other things (since this issue should not be blocking startup of the app nor causing swap usage limits to be exceeded).

[SidShetye]

@anurse We already have an open Azure support ticket for the 3rd item.

Of course, an ApplicationInsight release that resolving these errors/warnings would be great but till then we're running some tests over the weekend to have some data supporting/rejecting the idea that these are all related stability issues.

[flatproject]

I have fixed the

"Startup assembly StartupBootstrapper failed to execute. See the inner exception for more details. Could not load file or assembly 'StartupBootstrapper, Culture=neutral, PublicKeyToken=null'. The system cannot find the file specified. "

by uninstalling the Azure web app AI site extension as suggested here.

I have also fixed the

"Startup assembly Microsoft.AspNetCore.AzureAppServices.HostingStartup failed to execute. See the inner exception for more details. Could not load file or assembly 'Microsoft.AspNetCore.AzureAppServices.HostingStartup, Culture=neutral, PublicKeyToken=null'. The system cannot find the file specified. "

by installing "<PackageReference Include="Microsoft.AspNetCore.AzureAppServices.HostingStartup" Version="3.1.0" />"

Also applied the advice from #15381 (comment) but not sure if it helped.

Now the only issue i get is

"Startup assembly DiagnosticServices.HostingStartup failed to execute. See the inner exception for more details. Could not load file or assembly 'DiagnosticServices.HostingStartup, Culture=neutral, PublicKeyToken=null'. The system cannot find the file specified. "

which i can only fix by following the advice from #17370 (comment) which disables the profile and snapshot feature.

Is it possible to have profile and snapshot features working without getting the " DiagnosticServices.HostingStartup failed to execute" exception message?

I'm using framework dependant deployment from within visual studio publish screen and the app is .net core 3.1.0 64bit

[tstojecki]

Just to confirm, the steps outlined by @flatproject worked - Thanks!
Had to go through them all on a vanilla .net core 3.1.1, VS 2019 Publish, FDD.

Before arriving here, had to fix another issue in 3.1.1 with Microsoft.AspNetCore.Authentication.AzureAD.UI which was also crashing the app
#18401

All in all, 5 or 6 issues to resolve to get to a working 3.1 app through VS publish.

For this issue, specifically, I agree with the sentiment to reevaluate how these extensions affect the core service. At the minimum, if something goes wrong, they should not crash the entire app.

[SidShetye]

... which disables the profile and snapshot feature.

@flatproject On a related note, if you're on Linux, it's probably a good idea to disable Application Insights' profiling and snapshot because we're seeing an unrecoverable memory leak from it. See microsoft/ApplicationInsights-dotnet#1678

[SidShetye]

@davidfowl Is the purpose of Microsoft.AspNetCore.AzureAppServices.HostingStartup documented anywhere? And why does AppService throw a critical log message over it?

Seems like bad practice to include an entire binary (Microsoft.AspNetCore.AzureAppServices.HostingStartup) simply to mask a log warning when we need no functionality from it.

[davidfowl]

@SidShetye The site extension enables the logging options in the azure portal for azure app service to work. The HostingStartup is a way to plug into the application startup process without that code having be in the application initially. Think of it as a plugin. There's a plugin that runs on app service to enable app service features. When it breaks it's because there's a version mismatch, that is, there's a newer version of .NET Core than the extension supports (this is what we're working on moving forward, extensions are tied to the .NET Core major minor version).

You can enable the same feature by adding the package reference directly and calling the registration API (this is what the plugin package does) and why things work if you reference the package directly.

[jannikbuschke]

To be honest, these portal injected 'magic' extensions have rather painful in our experience. They don't feel production-ready (all the tickets), hurt testability (behavior changes post-testing), aren't well documented (like which env variables are injected from which extension? cleanup?).

+1

Azure app services adds many maybe well intented but overall counter productive things in the portal.

[mikerains]

I have fixed the

"Startup assembly StartupBootstrapper failed to execute. See the inner exception for more details. Could not load file or assembly 'StartupBootstrapper, Culture=neutral, PublicKeyToken=null'. The system cannot find the file specified. "

by uninstalling the Azure web app AI site extension as suggested here.

I have also fixed the

"Startup assembly Microsoft.AspNetCore.AzureAppServices.HostingStartup failed to execute. See the inner exception for more details. Could not load file or assembly 'Microsoft.AspNetCore.AzureAppServices.HostingStartup, Culture=neutral, PublicKeyToken=null'. The system cannot find the file specified. "

by installing "<PackageReference Include="Microsoft.AspNetCore.AzureAppServices.HostingStartup" Version="3.1.0" />"

Also applied the advice from #15381 (comment) but not sure if it helped.

Now the only issue i get is

"Startup assembly DiagnosticServices.HostingStartup failed to execute. See the inner exception for more details. Could not load file or assembly 'DiagnosticServices.HostingStartup, Culture=neutral, PublicKeyToken=null'. The system cannot find the file specified. "

which i can only fix by following the advice from #17370 (comment) which disables the profile and snapshot feature.

Is it possible to have profile and snapshot features working without getting the " DiagnosticServices.HostingStartup failed to execute" exception message?

I'm using framework dependent deployment from within visual studio publish screen and the app is .net core 3.1.0 64bit

I also had this mysterious "azure app svc Startup assembly Microsoft.AspNetCore.AzureAppServices.HostingStartup failed to execute" exception logged in App Insights, from Asp Net Core 2.1 on Windows, and was not able to make it go away with Self-Contained or Framework-Dependent. All the App Insights logging and TelemetryClient seemed to be working.
Errors are automatically logged from Controller Constructors, and even from exceptions during Startup. But seeing this Exception is innerving.

I found this issue and suggestion to install this package ... I can confirm that installing NuGet package Microsoft.AspNetCore.AzureAppServices.HostingStartup stopped the logged entries.

[clement911]

We also get this exception on every startup of our Azure app service.

I just upgraded .NET 5.0 and it is still happening.

What is the latest/recommended way of getting rid of this message. Are we really forced to install Microsoft.AspNetCore.AzureAppServices.HostingStartup to get rid of it?

(We currently don't use Application Insights)

[woeterman94]

Got the same issue when running a .NET 5 app in a docker container on azure app service.

[davidfowl]

cc @shirhatti @noahfalk

[samsmithnz]

Adding my vote to resolve this too. .NET 5 application, self-contained, deployed to a webapp. I seem to get the error on every deployment/startup. It doesn't seem to be breaking anything, except for filling up my logs with errors.

[BrennanConroy]

What's the TLDR? @davidfowl

[davidfowl]

It's that this issue has nothing to do with ASP.NET Core and @shirhatti will talk to the app insights team

[ghost]

Thanks for contacting us.
We're moving this issue to the Next sprint planning milestone for future evaluation / consideration. Because it's not immediately obvious that this is a bug in our framework, we would like to keep this around to collect more feedback, which can later help us determine the impact of it. We will re-evaluate this issue, during our next planning meeting(s).
If we later determine, that the issue has no community involvement, or it's very rare and low-impact issue, we will close it - so that the team can focus on more important and high impact issues.
To learn more about what to expect next and how this issue will be handled you can read more about our triage process here.

[clement911]

I'm confused to where we should track this moving forward but I just wanted to say it's still happening for us, on .net 5.0.
We managed to get rid of all startup exceptions we had except this one.

[davidfowl]

• StartupBootstrapper isn't part of .NET, it's part of Application insights.
• Microsoft.AspNetCore.AzureAppServices.HostingStartup - is our package and it's possible to run into similar issues with it loading. It usually happens when there's a version mismatch between the azure site extension and your application. At the moment , the easiest way to get rid of the error is to install the package and deploy it.

[adityamandaleeka]

Closing as a duplicate of #34864