Support app_offline.htm

[sayedihashimi]

I was chatting with @coolcsh @rowanmiller @davidfowl @danroth27 and a few others and we would like to enable support for taking the ASP.NET 5 application offline when a app_offline.htm file is present in the web root just like ASP.NET 4.5 supports.

We would like this primarily for the following scenarios.

• Take the app offline before EF migrations are executed
• Take the app offline when files are locked preventing publish

We would like to add a checkbox to the web publish dialog in VS allowing the user to indicate if app_offline.htm should be dropped. Web Deploy already has support for this, so on that side there shouldn't be much work.

[rubenprins]

I'm not sure why app_offline.htm should be needed "when files are locked preventing publish". ASP.NET should never, ever lock any files. If it does, ASP.NET is broken.

Come on guys, ASP.NET 1.0 had this solved over a decade ago (shadow copying).

Having support for app_offline.htm during an update is a welcome feature, however. But it should not be used to fix file locking issues that shouldn't exist in the first place.

[sayedihashimi]

@rubenprins I'll let the runtime guys respond back regarding file locking.

There are still scenarios where it makes sense to take the app offline. For example you are going to update the DB that the app talks or updating other apps that it depends on. It's not just about file locks with publishing.

[rubenprins]

@sayedihashimi, like I said: I agree, app offline is a useful feature for a multitude of reasons (just taking an application offline without hassle, for example).

But I was referring to the file locking issue you referred to in your second point. There's also a DNX issue regarding file locking preventing publishing (Azure was mentioned explicitly, but I'm afraid it affects any non-trivial hosting scenario).

So I'm starting to get a bit worried about the scrapped/backlogged feature list for vNext by now, if even features like this are backlogged.

[sayedihashimi]

But I was referring to the file locking issue you referred to in your second point. There's also a DNX issue regarding file locking preventing publishing (Azure was mentioned explicitly, but I'm afraid it affects any non-trivial hosting scenario).

So I'm starting to get a bit worried about the scrapped/backlogged feature list for vNext by now, if even features like this are backlogged.

That is an issue we should raise at in the dnx. I noticed there are some issues already logged like aspnet/dnx#1393 but I'm not sure it's exactly what you are referring to.

[PureKrome]

Curious question: would it be possible to make the filename configurable? totally low priority, etc. but just curious - like a possible backlog item for the future... 🙏 (eg. if yes, i'd like to look at submitting a PR for this)

[davidfowl]

We're not shadow copying anything, it has been a source of pain in the past that we want to avoid in the future.

[rubenprins]

@davidfowl, I agree, shadow copying is not without its flaws. And preventing updates from happening in the first place, you will prevent a whole slew of other issues at one fell swoop!

Like not restarting after config changes, app_offline support, web farms, msdeploy, etc.

Or will we be getting "guidance" on how to work around these dropped features as well?

And are you planning on providing a good overview of important changes to the platform that aren't improvements, or changes for the sake of change, but deliberate feature cuts of functionality people might, I don't know, expect to just work like they always have.

[guardrex]

Moving over from dotnet/aspnetcore#476 ...

I agree with @rubenprins ... a smooth WebDeploy is critical in its own right regardless of whether or not app_offline works or not.

How about Web Deploy recycleApp Provider and described in an MSDN Blog?

Issue the recycleApp with a recycleMode=StopAppPool, perform the deploy operation, then issue a 2nd recycleApp with recycleMode=StartAppPool to get it back up. Would this get the IIS worker process to drop its lock(s) fast enough for the deployment step?

I guess if nobody knows I could do some testing and report back if that would be helpful. As it stands, at least 50% of my publish attempts fail when issuing a raw WD publish command.

[guardrex]

... and there's this posted in an IIS Blog:

App Offline Template

WebDeploy v3.5 adds an enhancement to the existing AppOffline rule which allows you to take an ASP.NET application offline before publishing. In this release, WebDeploy supports specifying the relative path to an app offline template file located on the destination system. This rule handler applies to the following providers: appHostConfig provider, contentPath provider and iisApp provider.

The app offline template file can capture any static HTML content to be displayed for users while the app is offline. Here is example usage:

msdeploy.exe -verb:sync -source:iisApp=sourceApp -dest:iisApp=destApp,appOfflineTemplate="offlineTemplate.htm" -enablerule:AppOffline

[guardrex]

Single Web Deploy command with AppOffline or recycleApp does not work

The -enablerule:AppOffline doesn't work with a physical path contentPath, as is required here. The exception that comes back is:

Error Code: ERROR_SITE_DOES_NOT_EXIST
More Information: Site 'F:' does not exist.

WD needs to know the IIS site name to kill its AppPool, which makes perfect sense.

However, we can't give it the site name: A physical deployment path is required because you must publish to the folder above the wwwroot folder. If you try to publish by IIS app name, it uses the physical path in IIS, which is to the wwwroot folder, thus WD will deploy the approot and wwwroot folders to the wwwroot folder on the server ... of course, that breaks the directory structure of the project. Since you can't specify the IIS app name for AppOffline to work and simultaneously provide a physical path that doesn't match IIS's path for the site, AppOffline looks like a dead end ... at least insofar as trying to do it all in one command.

Same deal with recycleApp: It requires the destination to be an IIS app name. Besides, you can't add recycleApp in a single deployment operation that will take the AppPool down before deployment and put it back up after the publish. It's setup to either take it down, start it, or recycle it.

Separate Commands with recycleApp may be working OK

Splitting out into three steps seems to be working, but I'll need to do this for a little while to see how reliable it is. My only concern is that the IIS worker process will not release the lock in time for the deployment step, but I guess adding a delay would be an option if I see that is happening.

I start by issuing a recycleMode=StopAppPool using the IIS site name:

"C:\Program Files (x86)\IIS\Microsoft Web Deploy V3\msdeploy.exe" -verb:sync -allowUntrusted -source:recycleApp -dest:recycleApp="IIS_SITE_NAME",recycleMode="StopAppPool",ComputerName="https://CLOUD_SERVICE_NAME.cloudapp.net:PORT/msdeploy.axd",UserName="USERNAME",Password="PASSWORD",AuthType="Basic"

I use my normal deployment command next with the correct physical path for the deployment. Note that the physical path on the server is the folder above the wwwroot folder. For example, if I have my IIS physical path set to G:\mysite\wwwroot for an app, then I would be deploying with WD to G:\mysite.

"C:\Program Files (x86)\IIS\Microsoft Web Deploy V3\msdeploy.exe" -verb:sync -allowUntrusted -source:contentPath="OUTPUT_FOLDER_PATH" -dest:contentPath="SERVER_FOLDER_PATH",ComputerName="https://CLOUD_SERVICE_NAME.cloudapp.net:PORT/msdeploy.axd",UserName="USERNAME",Password="PASSWORD",AuthType="Basic"

I finish with recycleMode=StartAppPool using the IIS site name:

"C:\Program Files (x86)\IIS\Microsoft Web Deploy V3\msdeploy.exe" -verb:sync -allowUntrusted -source:recycleApp -dest:recycleApp="IIS_SITE_NAME",recycleMode="StartAppPool",ComputerName="https://CLOUD_SERVICE_NAME.cloudapp.net:PORT/msdeploy.axd",UserName="USERNAME",Password="PASSWORD",AuthType="Basic"

I'll roll these into a script and add a short delay between commands and report back ... unless you think I've made a big 💩 here! I see @sayedihashimi dealt with these issues in-depth years ago.

[rubenprins]

@guardrex, Recycling or stopping the application pool might not be what you want in a production environment. If you're hosting more than a few applications (note: not just sites, but applications), you're more than likely to group applications in only a few application pools.

(Ten applications running in a single desktop CLR are almost guaranteed to be more scalable/cloud friendly than running ten Core CLRs in parallel, each with their own private assemblies, private JITed code and private garbage collectors.)

Besides, updating application A should never take down applications B to Z, IMHO. But this does appear the way ASP.NET v5 is going: kill the application host to update anything, be it .dlls, or config settings.

Remember when xcopy/bin-deployment was a selling point in v1? Compared do writing/deploying COM components for a web server before it (always requiring a restart). This was considered a deployment nightmare, which is why shadow copying was invented in the first place (warts and all). We're coming full circle here, and it's not an improvement.

Anyone wants to try this in an automated fashion in a webfarm? Every single update? I don't fancy telling our Operations department that every single update might succeed, depending how fast the apppool can shut down. Or not.

(/rant)

Please, someone from the ASP.NET team tell me I'm wrong or missing something here.

[davidfowl]

Recycling or stopping the application pool might not be what you want in a production environment. If you're hosting more than a few applications (note: not just sites, but applications), you're more than likely to group applications in only a few application pools.

Moving forward, we're only going to support a single application per application pool. In fact CoreCLR is this way already for ASP.NET 5 applications.

Besides, updating application A should never take down applications B to Z, IMHO. But this does appear the way ASP.NET v5 is going: kill the application host to update anything, be it .dlls, or config settings.

Yes, the host process has to die. Even if we shadow copy, the process still has to be cycled. We're not using appdomains anymore (and can't on CoreCLR).

Remember when xcopy/bin-deployment was a selling point in v1? Compared do writing/deploying COM components for a web server before it (always requiring a restart). This was considered a deployment nightmare, which is why shadow copying was invented in the first place (warts and all). We're coming full circle here, and it's not an improvement.

There are ways around this that we need to explore. Like file renames + a way to recycle the application from outside of it (app offline style or maybe via a signal?). I'm hesitant to just do what we had before because of the number of issues that arise not with just shadow copying but also the file watcher and reliability on environments like azure where it barely works.

Don't get me wrong, I think it's important to be able to deploy a new version of the site without failure. I just think there might be better ways of going about it. If all else fails, we know how we've been doing it for the last 10 years but I rather not start there.

[guardrex]

The three commands that I show above with recycleApp work reliably. I've been publishing for the past few weeks OK. I haven't run into locked files on the server with this approach.

Anyone following this approach/hack keep in mind:

1. Set your IIS website physical path to the wwwroot folder on the server.
2. Set your Web Deploy path to the folder above the wwwroot folder for the msdeploy.exe command I show above.

[rubenprins]

Moving forward, we're only going to support a single application per application pool. In fact CoreCLR is this way already for ASP.NET 5 applications.

Are you serious? How is that supposed to scale? We run IIS servers with 200+ applications. Mostly small-ish applications in virtual directories so we can keep the number of domains manageable (which is also an issue in itself with v5 with its split wwwroot/approot). With 200+ CLRs running, 90% of server memory would be DNX overhead alone! Forget about not being able to update/deploy: we can't even run them without OutOfMemoryExceptions 😲

And why are these design decisions not clearly documented anywhere? Where's the "Caution: features removed from ASP.NET because we didn't think they were pretty" topic?

[davidfowl]

Are you serious? How is that supposed to scale? We run IIS servers with 200+ applications. Mostly small-ish applications in virtual directories so we can keep the number of domains manageable (which is also an issue in itself with v5 with its split wwwroot/approot).

200 sites running in how many app domains? Are these sites isolated security wise? If so I'm sure you know that we even recommend ASP.NET 4.x sites don't use app domains for security isolation. There's also an interesting thread on the CoreCLR repository that discusses this (https://github.com/dotnet/coreclr/issues/642), feel free to add your scenario there.

If there's no security problems, how much isolation do you need? Statics? Do you actually share anything or are you just taking advantage of the sharing you get from using the same assemblies across different applications within the same process (this includes JITTEd code as well)?

With 200+ CLRs running, 90% of server memory would be DNX overhead alone! Forget about not being able to update/deploy: we can't even run them without OutOfMemoryExceptions 😲

I can't say for a fact if that's what would actually happen. For sure more processes are heavier but I'd love to try and replicate what you're doing to see how big of a problem it really is and if there are different mitigations for it. For example, if you made native images for those applications (the ngen equivalent on CoreCLR called crossgen) and loaded them into separate processes you would still benefit from sharing. We also have a feature that allows you to put a set of common packages (also can be crossgened) on the machine so that all applications share them (I promise it's not like the GAC 😉)

And why are these design decisions not clearly documented anywhere?

We're in preview still and some of the design decisions are tentative waiting user feedback to see what kinds of scenarios are missing. AppDomains are just gone as a concept but maybe CoreCLR introduced AssemblyLoadContext to do assembly load isolation without the marshalling burden (as an example of a scenario that was once solved by appdomain that can be solved in a much better way IMO now). It would be more constructive to compare the density between the current CLR and the CoreCLR and if it really is much worse and cannot be mitigated in practice then we need to do something about it.

Where's the "Caution: features removed from ASP.NET because we didn't think they were pretty" topic?

As we get closer to the final product, expect to see more docs appear on our main docs site (http://docs.asp.net/). In the past you would never seen any real time development or discussions happening and you couldn't even give feedback until a beta or released product was out. Now we give you access to daily builds, commits, discussions and we have to pick the right things to solve in order to ship a product. ASP.NET 5 isn't version 5, it's version 1 despite the naming.

That said, I do think it would be valuable to call out some of the major decisions even if they are tentative as it causes great discussions like this to happen with people in the real world 😄

[rubenprins]

Although this discussion is veering wildly off topic, just a few clarifications: no, we don't need security isolation; the applications we host are built by ourselves, so there's no trust issues.

And since most of the applications are still web forms, we are indeed profiting from a lot of (ngen'd) code and metadata sharing, as well as other memory/resources/(GC) threads the CLR deems fit to share between AppDomains. Also, a lot of applications are used only infrequently, so the process model idle timeout helps reduce memory as well, without unloading the entire ASP.NET runtime.

Only a year ago we made the effort to consolidate our application pools, and it really helped in reducing our memory usage.

Because most of our older applications are web forms based, without a lot of application-level caching and comparably little user code (BCL + System.Web + System.Web.Extensions will usually dwarf our codebase), so we benefit quite a lot from the sharing by the framework.

This is, of course, not the case for our MVC applications, where we haven't bothered to put the dlls in the GAC, as this proved too big a hassle for too little benefit.

Maybe crossgening will help us here, as long as it's discoverable, and doesn't require as much hand holding as deploying ASP.NET v5 applications currently appears to entail. (Which is one of the major points regarding this issue, as well as aspnet/dnx#928 for our scenario.)

IIRC there was/is a tool for ASP.NET v4 to enable sharing of private dlls by replacing them with symlinks and let the OS handle the sharing; but it's so well hidden it might as well not exist at all. It also doesn't really help with sharing JIT'ed code when you're using separate processes/application pools, but it could help the current MVC framework's footprint when consolidating code in a single application pool like we do.

[djanosik]

Is there any solution yet? I would like to deploy my project to Azure, but:

• I can't use WebDeploy. It fails because files are being locked and app_offline.htm is not supported.
• I can't use recycleMode=StopAppPool because it's not allowed to stop pools on Azure.
• I can't use Git CD because repository is not hosted on GitHub / Bitbucket.

I am running out of ideas.

[muratg]

@djanosik Did you try this? You can deploy from a local repository with git, and you need to host your app on GitHub or BitBucket.

[djanosik]

@muratg No :) I am sure it will work, but I need to setup continuous deployment.

[muratg]

It's a one time setting for your web site. You'll be basically creating a git endpoint and credentials for your site, and add that as a remote to your local git repo. Whenever you want to deploy you just say git push azure master.

[guardrex]

@djanosik Can you control the Azure Web App's VM with PowerShell? ... Do they allow that level of access to the underlying VM?

[djanosik]

@guardrex Well, I am not sure, but I will try. It's definitely good idea. Thank you!

[guardrex]

@djanosik You could try taking some bits from my experimental PS script and using PS to control your AppPool: https://github.com/GuardRex/net5-iis-ps-publish

Get a PS Session to the VM: https://github.com/GuardRex/net5-iis-ps-publish/blob/master/net5-iis-ps-publish.ps1#L238-L249
Stop the AppPool: https://github.com/GuardRex/net5-iis-ps-publish/blob/master/net5-iis-ps-publish.ps1#L389-L396
Restart the AppPool: https://github.com/GuardRex/net5-iis-ps-publish/blob/master/net5-iis-ps-publish.ps1#L439-L446

[brent-russell]

@djanosik There is a new MSDeploy provider that handles publishing asp.net 5 websites. If you perform a publish to Azure from Visual Studio and examine the output, you can see the command and parameters being used. It basically just wrangles the wwwroot and approot content going into different folders.

https://azure.microsoft.com/en-us/blog/web-deploy-3-6-beta-released/

"%env.MSDeploy%" ^
%system.TestWebDeployment% ^
-verb:sync ^
-source:IisApp='%teamcity.build.workingDir%\artifacts\wwwroot' ^
-dest:IisApp='%system.Deployment.SiteName%',computerName='https://mysite.scm.azurewebsites.net/msdeploy.axd',userName='%system.Deployment.Username%',password='%system.Deployment.Password%',includeAcls='False',AuthType='Basic' ^
-enableLink:contentLibExtension ^
-useChecksum

[djanosik]

Thank you for your suggestions. I've built a little helper that is now part of the FAKEX build script. It can stop / start Azure WebSites and deployment slots.

[haydenmc]

Just adding that I'm also experiencing pain here deploying to Azure Websites.

Web Deploy cannot modify the file 'BoinkChat.dll' on the destination because it is locked by an external process.  In order to allow the publish operation to succeed, you may need to either restart your application to release the lock, or use the AppOffline rule handler for .Net applications on your next publish attempt.
Learn more at: http://go.microsoft.com/fwlink/?LinkId=221672#ERROR_FILE_IN_USE.

I'm going to have to resort to finding a way to script shutting down the Azure Website before deploy, and starting it back up after.

[davidfowl]

We're going to tackle this problem.

@haydenmc is BoinkChat.dll your application dll?

[haydenmc]

@davidfowl Yup!

https://github.com/haydenmc/BoinkChat

[muratg]

@davidfowl

[guardrex]

You can script with msdeploy. Search "msdeploy" to find my example script,
or I'll post the link when I get back to the office.
On Nov 23, 2015 12:49 PM, "Murat Girgin" [email protected] wrote:

@davidfowl https://github.com/davidfowl

—
Reply to this email directly or view it on GitHub
#141 (comment).

[guardrex]

You can also script with PS ... see my repo for an example.
On Nov 23, 2015 12:54 PM, "Luke Latham" [email protected] wrote:

You can script with msdeploy. Search "msdeploy" to find my example script,
or I'll post the link when I get back to the office.
On Nov 23, 2015 12:49 PM, "Murat Girgin" [email protected] wrote:

@davidfowl https://github.com/davidfowl

—
Reply to this email directly or view it on GitHub
#141 (comment).

[sayedihashimi]

@davidfowl @muratg this will be needed more after we switch to producing binaries by default (RC2). Can you implement for RC2? If so we can enable the checkbox on the publish dialog.

[muratg]

I assume this (or something like this) is also required for HttpPlatformHandler interaction.

[sayedihashimi]

I think it's worse with HttpPlatformHandler yes

[guardrex]

@haydenmc I didn't realize this was the same issue where I posted earlier. The MSDeploy commands are above: #141 (comment) ... and they work quite well. You'll see a strange entry in the logs (aspnet/IISIntegration#32), but it won't affect your enjoyment of publishing if you go that route. The PowerShell commands are up there, #141 (comment), and you saw @brent-russell's hot tip: #141 (comment)

[muratg]

Bringing it back to RC2 so that we can discuss this in a design meeting.

[Tratcher]

Moved to aspnet/IISIntegration#81