Decompress frontend after pip install

[steverep]

The frontend wheel is currently approaching 50 MB; however, the actual size should be about half that because it packages both compressed and uncompressed versions of nearly every file. This creates wasted time on both the build/upload phase for releases, and the download/decompress phase for users. I believe the pyPI size limit is also still 60MB, so we may start flirting with that issue too.

My proposal is pretty simple:

1. Only package the compressed files when both exist
2. Optionally, skip wheel compression
3. Provide a method/script (in python) in the package to decompress everything. Core would then need to call this script/method somewhere after installing.

[frenck]

I believe the pyPI size limit is also still 60MB, so we may start flirting with that issue too.

We don't have to worry about those limits :) Which is kinda nice (the rest of your story still holds though!)

For the time to build, I can see why that is less nice, however, I think it is actually pretty nice we spent that time upfront instead of after installation? The "the download/decompress phase for users", isn't a real factor here, as we do that in our resulting Docker images already, so 96% of the users aren't affected by that.

It does make me wonder, if the webserver of HA should just handle it on the fly (with a caching mechanism)

[steverep]

Didn't realize we already had an exemption on the limits 👍🏻

however, I think it is actually pretty nice we spent that time upfront instead of after installation?

Not really, at least in theory. If the wheel is halved, that's about half the time to download and half for pip to decompress the wheel and install it. Theoretically, the time to decompress manually afterwards would be similar to what pip has to do, so overall time would be less.

It does make me wonder, if the webserver of HA should just handle it on the fly (with a caching mechanism)

Dynamic compression should really only be done for SSR or similar tasks. Recommendations are always to compress first whenever possible. Also, aiohttp only supports standard gzip. We use a Zopfli variant which gives better compression but takes much longer.

[frenck]

Not really, at least in theory. If the wheel is halved, that's about half the time to download and half for pip to decompress the wheel and install it. Theoretically, the time to decompress manually afterwards would be similar to what pip has to do, so overall time would be less.

That is not a user end... that was my point.

[steverep]

That is not a user end... that was my point.

Fair enough. I'd still like to implement this though to potentially speed up those image builds, but more importantly to cut down on the size of the workflow artifacts.

I'm working on some frontend testing stuff that will require bundles to be saved for each PR, so I need to do most of this implementation anyway to keep repo storage from getting out of control.

Where would the call to decompress need to go for core?

[balloob]

I think the big question is … what browsers still use the uncompressed version?

Look at MDN and it was part of so many first releases of browsers that instead of a version, for many the answer is simply "yes" 😅

Uncompressing on the fly seems reasonable.

[steverep]

You're absolutely right of course… 99% or more of users have absolutely no need for the uncompressed versions. I'd be perfectly fine with compressing everything and throwing the rest away.

The last time we discussed this, I think your only concern was with anyone having a Proxy setup that did its own compression. Creating a simple script to decompress. Everything is trivial so there's no need not to do that to satisfy those edge cases. If there is a way to decompress only when one of those edge cases specifically asked for it, we should do that and save everybody else some storage.

[steverep]

I should have also put in the OP that this proposal stems from discussion with @balloob and @bramkragten back in home-assistant/frontend#15663. Source maps are left uncompressed mainly because of the 60MB limit (and whether they are or aren't is obviously not as important as the actual sources).

[steverep]

Frontend PR is here: home-assistant/frontend#19838

Always decompressing after install is currently the only option because aiohttp will serve a 404 otherwise (aio-libs/aiohttp#8182). Core is currently overriding this class handler (for reasons I do not understand), and perpetuating the same issue.

cc: @bdraco