-
Notifications
You must be signed in to change notification settings - Fork 116
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
App Installer emitting a "Unknown error (0x80D05011)" on some machines #188
Comments
So an update on this. We had the customer reboot the machine. We then tried to install the app again but this time the App Installer was producing error code of 0x8000FFFF (which is the same as #186) however it had a different error text which did not say "Catastrophic error". We then tried the Add-AppxPackage workaround and hey-presto it worked, the app was successfully installed. There are a lot of weird bugs in the App Installer architecture... |
I am seeing this error frequently as well, and have found it difficult to diagnose. |
Any word on possible solutions for this? |
Closing this. Please reopen if you are still seeing the issue. |
@samiranshah Sorry I missed this. Yes I saw this issue ("Unknown error") again just a couple weeks ago. A reboot of the machine fixed it. |
We saw this issue today. A reboot did not fix it, unfortunately. |
@samiranshah Please reopen as we are also seeing this after changing the version number from 1.0.xx.yyy to 1.1.0.0 A reboot did fix it here. |
I've spent a few hours debugging this. Here's what's going wrong and how to work around it. This looks like a (very serious!!) bug in the MSIX / Delivery Optimization subsystem of Windows. It appears Microsoft already know about it and have fixed it back in May, but haven't yet released the fix, at least not to Windows 10. @wcheng-msft @samiranshah You guys need to document the bug and publish an official workaround if you aren't going to push out the fix ASAP because this is a nightmare to figure out, yet you clearly already know what the bug is! That's really not a satisfying discovery. The Delivery Optimization subsystem that is used by App Installer appears to be incorrectly caching the size of the file given a URL, in memory. If an
@yowl This is why you see the failure once you changed the version number. The problem is not actually the change of version number but rather that the new version quad is a different size textually to the previous. The situation may be complicated by HTTP redirects. I saw this problem myself before ever changing anything about the XML, and I think Windows cached the wrong size from the body of a 302 redirect from HTTP to HTTPS. So watch out for that if your server generates a lot of HTML when issuing redirects. You can see that this bug is happening by dumping the Delivery Optimization subsystem logs: Here's a stupid hack that seems to work:
(edited for clarity) |
Thanks @mikehearn . Here's some powershell that I'm putting into my pipeline to do as you suggest $size=(Get-Item "$($args[0])").length
Write-Host $size
$padSize=2000 - $size
Write-Host padding with $padSize
$padString = "".PadRight($padSize)
Add-Content "$($args[0])" $padString -NoNewline |
@samiranshah Please reopen this. this has been happening for over 1.5 years. It's a shame the way MS has been dealing with this. |
@mikehearn @yowl |
@mvelayati |
@yowl Yeah I uninstalled whatever had installed on my PC, installed the app again with the new fixed size appintaller. After that I ran the pipeline again to upgrade the version number and still no luck! |
It does seem to be holding up here. You've checked the size of the deployed file? |
yeah I set the file size to be exactly 2000 bytes using the powershell script you posted above and then I FTP it over to my host. |
Thanks for this fix idea. I'm going to put it in place a few weeks in advance of our next UWP rollout. That way it should give plenty of time for machine reboots and the cache to get cleared out. So by the time our update goes out - there shouldn't be any machines holding onto the unpadded cached copy. |
@mvelayati Not sure what else to suggest. Have you got redirects as @mikehearn mentioned? If you ftp the file back locally and check it, it is 2000 bytes longs? If it starts failing here, I'll report back that the script doesn't work. |
@yowl |
If the DO logs show a request for the exact same size as on the server, and it still fails, you might be facing a different issue then. What is the exact error you get? |
I just added 20kb of whitespace tail padding to our .appinstaller files. Expecting this to reliably fix these errors we were getting a lot:
It's genuinely crazy that such a critical piece of Windows infrastructure contains such a catastrophic caching bug and that it has gone unfixed for so long. I imagine this bug must affect hundreds of other components aside from just App Installer. |
I suspect App Installer is probably the only thing seriously affected. If it affected Windows Update it'd have been caught and fixed long ago, but it's obvious that Microsoft isn't actually using AppInstaller internally. Hypothesis: Delivery Optimization originally made the assumption that all static assets it'd be asked to download were immutable. When AI integration was added this assumption was violated but nobody noticed. The symptoms probably don't appear if you test by simply incrementing a version a few times, because until you do > 9 releases the version number won't change size, and the errors it generates can vary, and it's a caching bug so it will seem to come and go more or less at random. Overall hard to track down. However from another GitHub thread Microsoft mentioned that they already fixed it back in May and were looking at maybe backporting the fix ... clearly that never happened, which is what's really crazy here. The bug is guaranteed to eventually hit everyone using AppInstaller, yet, MS don't seem to care. |
@mikehearn Priceless, and probably true 😁 |
@samiranshah Can you please reopen this issue? We still see this issue regularly. |
Found this in the Get-DeliveryOptimizationLogs command from a few days ago. Relates to the failure to download the .appinstaller file. Same error code. As you can see the Blob Storage is responding with a Partial Content and Content-Range so undoubtably it does support it. DoSvc is screwing up somewhere. If you look closely it has requested 1 byte less than it should have to get the complete file. Bizarre? |
I've altered my padding script such that it will always pad until the file size = 32289 bytes (as per DoSvc logs above, which is what it expected my file size to be - but actually was not). I believe just blindly adding 25K of whitespace to the end was insufficient as a fix as ultimately the file size is still going to change and make DoSvc freak out. Though it was sufficient to get around the XML parser errors (which were another failure mode). |
Yeah that'll be it. My code is calculating the right amount of whitespace to ensure the file is always a constant size. I believe it's insufficient for the file to change size but only within the whitespace area, because DO notices the data it got back is a different size to what it expected and pukes, as you found. |
Hi all, can we get logs via Feedback hub again? the current logs have expired and we need them to investigate. |
Hi @dianmsft are the DoSvc logs above not already useful enough? The DoSvc is attempting to download a file using invalid file size information from its in-memory cache. |
@dianmsft Can I refer you to this post - #188 (comment) - by fellow contributor @mikehearn which details the precise issue and is surely more than enough information to be able to investigate, reproduce and bug fix the issue. My DoSvc log output above may also be of use. If you look closely the logs show that DoSvc think the file size is 32289 bytes but this was incorrect. It made that assumption from its in-memory cache. In principle I suppose there is no harm in caching that information - but the way it handles the unexpected condition is where it goes really wrong. It basically just gives up at that point because one of its assumptions (file size) was wrong. It shouldn't give up. It should just retry a basic GET request without any specific Content Range Request. As @mikehearn detailed in his post, it seems like DoSvc was designed for use with CDNs serving immutable file content. This is even alluded to in my DoSvc logs because it makes reference to "cdnUrl". But the App Installer architecture makes no such assumptions (nor does it document this implementation detail) that the .appinstaller file is assumed to be immutable. An alternative solution would be to not wholly rely on DoSvc as part of the App Installer architecture. If DoSvc is returning error codes - or is just plainly "not available" (the user might have stopped the DoSvc Windows Service) - then App Installer really ought to try bypassing DoSvc entirely and issuing the HTTP requests itself. This would massively increase the robustness of App Installer against these edge-case error modes. After all DoSvc is only meant as a bandwidth saving optimisation. If an optimisation is - for whatever reason - not available or not working as expected then just forego attempting to apply the optimisation. |
@dianmsft You need to start copying logs out of the Feedback app into your internal bug tracker as soon as they're submitted. This isn't the first time you or a colleague has come back months later on a thread where people report critical errors and say "oops we lost the logs". You're acting like this is a routine problem - it isn't. I've never experienced before a company requesting error logs from a customer, doing nothing with them, months later admitting the logs were thrown away without being looked at and then asking for the user to repeat the work again. We're making a purely voluntary effort to help you here. We aren't your colleagues, nor your reports. Our time has value, and when you make requests like this it's clear that your team has developed a culture of viewing customer time as free. MSIX is a great technology when it works, but your attitudes are making Microsoft look like a totally unreliable partner. |
@nbevans The last information we have on this from actual Windows developers is that the bug was fixed some time ago, and the bug fix has shipped already. I haven't tried rolling back my fix, but is it possible your VMs or machines aren't fully patched? Above I can see you're running Windows 10 for instance, does it reproduce on Windows 11? |
Hmm. I don't know why you wouldn't be offered the latest update. Maybe a hardware issue? I have a nasty sinking feeling that maybe Microsoft don't intend to backport the bug fix to the version you have? @dianmsft Could you please confirm with the relevant engineers who wrote the fix exactly what versions they backported it to, so we know what versions of Windows will work and which need the workaround? If you ask the engineering team about this bug they will know what it is and when they fixed it. |
I'm running Win10 on Parallels on my Mac. It's about as "clean room" environment as one could ever expect. Normally this kind of setup doesn't present any reasons to block offering Windows Updates. |
You can force the upgrade to 21H2 via the Windows 10 Upgrade Assistant: https://www.microsoft.com/software-download/windows10 if you want to give it a shot (maybe take a snapshot of the VM, upgrade it, then restore the snapshot)? |
I have applied the 21H2 update but I still have the error. Is it possible that it's only fixed for Windows 11 ? |
It is certain. I am developing an application and I can reliably reproduce this any time, as I have been able to since I first started using msix almost 2 years ago. I have 21H2. The problem is in the delivery optimisation system. Given how microsoft have dragged their feet on this issue till now, I doubt it will ever be fixed on windows 10. Glad to hear it at least works on 11. |
tl;dr |
Just reported by yet another user of our app.. |
Just experienced this issue on a Windows 10 21H2 after spending a few days learning this new "improved" installer/package delivery system from Microsoft. Reminds me of the ClickOnce/SmartScreen fiasco a few years ago: https://robindotnet.wordpress.com/2013/02/24/windows-8-and-clickonce-the-definitive-answer-2/ that if memory serves well also took Microsoft several years to fully address. @dianmsft please commit to get this bug fixed on Windows 10 ASAP!!!! We need a robust installer/package delivery system for Windows 10 too!!!! |
@dianmsft may I remind you that Windows 10 Retirement Date is Oct 14, 2025, in other words, in more than 3 years! Hence this bug needs to be fixed in Windows 10. When can we expect a Windows 10 bugfix? |
How does this bug still exist for 3 years in a system used to deliver software to clients? |
We've been asking for that for years with no success, it's presumably some higher level policy to do with Windows 10 backports that nobody on the team is motivated to try and escalate. Or they're ignoring this ticket. Or both. There is a fix though. If you're against a bit of product hawking, look away now. My company makes Hydraulic Conveyor which can produce MSIX packages with all the necessary workarounds for Windows 10. It pads the .appinstaller file, it verifies you're not about to change an already released version, it also provides a small stub EXE that bypasses the App Installer UI app entirely and drives the install via the package manager API. This stub EXE is then used to work around/improve on other MSIX aspects, for example, it reimplements synchronously checking for updates on launch to be faster and more reliable. Conveyor is mostly targeted towards people distributing cross platform apps, but there's no reason it can't be used for Windows only apps too. It's configured via a simple JSON superset, and has a few other features that might be interesting to people on this thread:
Finally, for small quick packages where you're just wanting to distribute something internally without hassle, the next release (coming out in a few days) can draw icons for you too. It's a local tool you run yourself, it's free for open source projects and paid otherwise. I realize that's not as good as actual fixes from MS but I hope it may help some people. This one is by far not the only bug in Windows, so having a tool that abstracts you a bit from the the OS will often be quite handy. |
It is more like a feature now, not a bug, if even some packaging/release management software includes workarounds for this. Have App Installer reached version π already? 🤣 |
I stopped using the App Installer app for updates long time ago. It will only be used to initially install our app. Updates are managed by our app using the packaging API's. |
Hi 👋 , we've just hit this issue (file size differences in appinstaller file breaking Windows 10 updates). |
@MarkIngramUK Before rolling your own set of workarounds definitely check out Conveyor for your apps. It does use the PackageManager APIs to do the initial install and can also use them to do updates, whilst also using the |
I can confim that MS hasn't bothered to fix this in Windows 10 as of 25/05/2023. I am currently on Windows 10 Business 22H2 19045.2965. Trying to leave ClickOnce behind and move to MSIX. Update management is a deal breaker so to complete the move need this working. Restart the PC is the de-facto workaround without using any of the clever stuff described above (e.g., padding the file), everytime I restart, it works. |
The OS servicing update should be coming on Windows 10 22H2 Build 19045.3030 (KB5026435):
|
@florelis That's great news! Thanks for getting this backported. We look forward to being able to remove our workarounds one day. |
It's been a year with no further complaints here, so presumably by now the backport is widely distributed enough for it to be considered fixed (I haven't tested). Therefore @dianmsft please close this bug (as I can't). |
Similar to #186 - but a different error code. Also, with this error code - the workaround to use Add-AppxPackage via PowerShell does not work. So the .appinstaller cannot be installed at all on machines that exhibit this issue - we are unsure currently what to advise to the customer short of "try reinstalling your machine from scratch".
Here is a Feedback Hub link to captured diagnostic data when attempting to install via App Installer: https://aka.ms/AAa0oga
The text was updated successfully, but these errors were encountered: