Supporting ETag http headers on static files

[mathertel]

The ETag (or entity tag) HTTP header specified the version of the returned resource.
This is done by the default implementation by calculating the md5 checksuom of the file content.
A custom ETag calculation can be passed as an option.

This allows client side caching to be more efficient and save bandwidth.

The implementation is taken from esp8266/Arduino and is compatible with the WebServer implementation for esp8266.
where it was implemented by @imwhocodes and me (@mathertel).

The WebServer now include the methon enableETag(bool enable, ETagFunction fn = nullptr)
to enable and disable ETag Headers for a server.

The StaticRequestHandler calculates the ETag value when enabled and checks / populates the header.

An optional ETag calculation function can be passed to enableETag (e.g. to provide different algorithms).

Example

The WebServer Example illustrates using the ETag function.

It is using the "last change date" of the filesystem to create ETags wich is more efficient.

It is available on some file systems implementations (LittleFS, FSFAT but not SPIFFS)
and requires NTP to provide a current datetime.

Some Doku

about http ETags

ETags are "hints" delivered with a web response in the ETag http-Header to the client.

This hint value is added to further requests to the same resource in the .If-None-Match Header.
Hereby the server can check if the client should still use the version in the cache or needs an update.

The web server does not send a full response and file content when the content was not changed
but just returns a 304 Not Modified status to the client resulting in much smaller network packages.

In the cases when the clients will request for the same resources multiple times in a row
the server-side checksum check is a lot more effective than transferring the bytes over the network.

As this is not always true there is an extended option to inject a function that behavior can be customized and tailored to the specific requirements.

enabling ETag support

To enable this in the embedded web server the enableETag() can be used.
(next to enableCORS)

In the simplest version just call enableETag(true) to enable the internal ETag generation that calcs the hint using a md5 checksum in base64 encoded form. This is an simple approach that adds some time for calculation on every request but avoids network traffic.

The headers will look like:

If-None-Match: "GhZka3HevoaEBbtQOgOqlA=="
ETag: "GhZka3HevoaEBbtQOgOqlA=="

ETag support customization

The enableETag() function has an optional second optional parameter to provide a function for ETag calculation of files.

The function enables eTags for all files by using calculating a value from the last write timestamp:

server.enableETag(true, [](FS &fs, const String &path) -> String {
  File f = fs.open(path, "r");
  String eTag = String(f.getLastWrite(), 16);  // use file modification timestamp to create ETag
  f.close();
  return (eTag);
});

The headers will look like:

ETag: "63bbaeb5"
If-None-Match: "63bbaeb5"

[VojtechBartoska]

Thanks for nice and complete PR @mathertel! We'll review it soon.

[PilnyTomas]

@mathertel nice work!
Would it be easy to add some extended feedback on the file upload? For example in serial output mention the file name and size and/or enable downloading it back.
Also please modify the .ino header and README.md so that it is using our new template
Otherwise LGTM.

[P-R-O-C-H-Y]

Any updates @mathertel on @PilnyTomas 's comment?

[mathertel]

• used provided template for readme
• TRACE output on file upload as requested.
  please review.

[PilnyTomas]

@mathertel Thanks for the addition. Please make sure that when someone tries to upload a too large file it doesn't run away with a looped message:

./components/esp_littlefs/src/littlefs/lfs.c:584:error: No more free space 385
./components/esp_littlefs/src/littlefs/lfs.c:584:error: No more free space 385
./components/esp_littlefs/src/littlefs/lfs.c:584:error: No more free space 385
And to the infinity ...

For small files, the messages are enough.

[mathertel]

In the readme:

File upload

By opening http://webserver/$upload.htm you can easily upload files by dragging them over the drop area.

Just take the files from the data folder to create some files that can explore the server functionality.

Files will be uploaded to the root folder of the file system. and you will see it next time using http://webserver/files.htm.

The filesize that is uploaded is not known when the upload mechanism in function
FileServerHandler::upload gets started.

Uploading a file that fits into the available filesystem space
can be found in the Serial output:

starting upload file /file.txt...
finished.
1652 bytes uploaded.

Uploading a file that doesn't fit can be detected while uploading when writing to the filesystem fails.
However upload cannot be aborted by the current handler implementation.

The solution implemented here is to delete the partially uploaded file and wait for the upload ending.
The following can be found in the Serial output:

starting upload file /huge.jpg...
./components/esp_littlefs/src/littlefs/lfs.c:584:error: No more free space 531
  write error!
finished.

You can see on the Serial output that one filesystem write error is reported.

Please be patient and wait for the upload ending even when writing to the filesystem is disabled
it maybe take more than a minute.

[mathertel]

@PilnyTomas : All implemented as you suggested. Please Review or give some feedback.

[mathertel]

This is a compatibility request to ESP8266 to easy migration to ESP32 (C3).
Would be welcome to include it in the next version.
@PilnyTomas : All implemented as you suggested. Please Review or give some feedback.
@P-R-O-C-H-Y : Who can do the review when @PilnyTomas doesn't find the time ?

[VojtechBartoska]

hello @mathertel, thanks for the updates. We'll review your PR soon.

[PilnyTomas]

Hi, sorry for the delay, I had some happy changes in my life, so I took a few weeks off.
Thank you for your updates.
I would love to see a different message when the upload fails - instead of "finished" there could be something like "Upload failed"
It isn't necessary - just nice to have, so I approve the PR.

[CLAassistant]

All committers have signed the CLA.

[P-R-O-C-H-Y]

@mathertel Please add a empty .skip.esp32h2 file to your example folder to fix failing CI.

[mathertel]

@mathertel Please add a empty .skip.esp32h2 file to your example folder to fix failing CI.

done.

[lucasssvaz]

Small nitpick but LGTM

[lucasssvaz]

Suggested change

	</html>
	</html>

[me-no-dev]

I think you fixed index.htm (which was also to be fixed), but forgot to also add a line at the end in files.htm as well

[P-R-O-C-H-Y]

@mathertel Please fix the missing empty line at the end of files.htm as suggested and LGTM!