re-visit re-validating Web cache

[proddy]

With consistent newer versions of the WebUI some users still need to CTRL-R to browser refresh the cache. I was hoping we had fixed this by calculating a hash code for each html/js .gz file, including the date (which is the build time) and adding this to the HTTP headers, but it seems not to be fully working.

The code is in function ESP8266React::WWData::registerRoutes() with this code section:

            // Check if the client already has the same version and respond with a 304 (Not modified)
            if (request->header("If-Modified-Since").indexOf(last_modified) > 0) {
                return request->send(304);
            } else if (request->header("If-None-Match").equals(hash)) {
                return request->send(304);
            }

            AsyncWebServerResponse * response = request->beginResponse_P(200, contentType, content, len);

            response->addHeader("Content-Encoding", "gzip");
            // response->addHeader("Content-Encoding", "br"); // only works over HTTPS
            // response->addHeader("Cache-Control", "public, immutable, max-age=31536000");
            response->addHeader("Cache-Control", "must-revalidate"); // ensure that a client will check the server for a change
            response->addHeader("Last-Modified", last_modified);
            response->addHeader("ETag", hash);

Need to look into this again.

[tp1de]

I reopened #2166. The web-ui errors and api errors co-exists and might have the same root cause. Please check.

[proddy]

This issue is about web cache, not the API's failing. If you find an issue, please create a new one. I have removed the old comments to keep this feature clean.

[proddy]

I did some checking on an ESP32-S3 and v3.7.1-dev, and it does look like the cache is working. When I deliberately change some of the web code and re-upload using the Download/Upload page, the browser refreshes the cache for the changed .js files (HTTP 200) and displays the correct version without the need for a ctrl-r/ctrl-f5, while the ones that haven't been changed stay at HTTP 304.

This makes sense since the web code already uses Cache Busting during the webpack process. Each generated filename has a unique generated hash key in the name like

dist/assets/index-BVzJnzPI.css           0.51 kB │ gzip:   0.26 kB
dist/assets/vendor-CC-BRDYN.css         14.11 kB │ gzip:   2.68 kB
dist/assets/index-B9FgGMrO.js           11.51 kB │ gzip:   5.27 kB
dist/assets/index-Ba2DbYOs.js           11.74 kB │ gzip:   5.56 kB
dist/assets/index-CAHU6rdT.js           11.82 kB │ gzip:   5.57 kB
dist/assets/index-Do0A2NOF.js           12.25 kB │ gzip:   5.96 kB
dist/assets/index-DX_LDPGF.js           12.26 kB │ gzip:   6.01 kB
dist/assets/index-TC0xYsad.js           12.27 kB │ gzip:   5.62 kB
dist/assets/index-D-YBhaBX.js           12.31 kB │ gzip:   6.12 kB
dist/assets/index-Bd5IoDaM.js           12.55 kB │ gzip:   5.68 kB
dist/assets/index-ULlYCs_X.js           12.72 kB │ gzip:   5.89 kB
dist/assets/index-DMNTwoSo.js           12.82 kB │ gzip:   5.80 kB
dist/assets/index-DM_aTSgt.js           13.70 kB │ gzip:   6.67 kB
dist/assets/index-DKi9DpQC.js          194.59 kB │ gzip:  53.12 kB
dist/assets/vendor-CqciUFmp.js         471.35 kB │ gzip: 148.21 kB

I'll leave this issue open for a while. I only tested on Edge. If anyone can reproduce the browser refresh not happening, I'll gladly dig deeper into this.

For reference, this is a nice article: https://medium.com/@fatehalisulthoni/cache-busting-3-ways-to-fix-stale-deployment-in-react-8fc046fa7e92

[proddy]

I heard from BBQKees that some users are still seeing issues with the Web not refreshing after a firmware upgrade (browser cache is stale). If anyone encounters this issue please update this GitHub issue with

• the browser and version you are using
• any errors or strange anomalies you see in the webUI after the update
• if possible a screenshot of the F12/dev looking at the Network when the pages are loaded. The status code (200 or 304) is useful to see.

[MichaelDvP]

I think this is when updating from v3.6.5, 3.6.5 have cache setting expire after one year, so brower take it and does not look for new settings.

[bbqkees]

Usually I get these questions when people are upgrading from 3.6.5.
And sometimes they never updated their Gateway after purchase, so it may have been running for a year on <3.6.4 before they update for the first time.
So as the behavior is different for 3.7.0 onward, this cache problem will fade away over time when users have all updated above 3.6.5.

[proddy]

okay. I need to check in which version I added the Etags and cache timeouts. I thought it was in 3.6.5. The generated .js filenames should be different when going from 3.6.5->3.7.x so it's strange that they don't automatically refresh. We can certainly add some text to the "EMS-ESP is restarting, please wait..." but I'd rather find a more elegant way. Kees once suggested that when EMS-ESP starts, it'll show a Welcome Page (which you can click away) with upgrade notes. We can make a new Rest endpoint for this and load the text from the backend, but it'll mean adding all this static text into Flash for each version.

[MichaelDvP]

The js differs, but the html page has same name. Browser loads the html page, checks the cache and loads the old page and scripts.
It's one time Shift-CTRL-R to clear disk-cache entry and reload (CTRL-R only reloads).
If you want to add a hint, do it on a new filename, not a popup on existing page (that does not load in case of cached page).

[proddy]

The js differs, but the html page has same name. Browser loads the html page, checks the cache and loads the old page and scripts.

are you positive this is how it works? I'm seeing:

note the cached 304's