Propose TemplatePrinter

[Chris--A]

I have created a simple template processing tool. Primarily this has been built to work with PsychicStreamResponse proposed in #45. Incorporation directly into PsychicStreamResponse and other response types is possible. Implements some of #55.

This is not specific to PsychicHttp, and it works with any Print object. You could for example, template data out to File, Serial, etc....

The template engine is a Print interface and can be printed to directly, however, if you are just templating a few short strings, I'd probably just use response.printf() instead. Its benefit will be seen when templating large inputs such as files.

One benefit may be templating a JSON file avoiding the need to use ArduinoJson.

Before closing the underlying Print/Stream that this writes to, it must be flushed as small amounts of data can be buffered. A convenience method to take care of this is shows in example 3.

The header file is not currently added to PsychicHttp.h and users will have to add it manually:

#include <TemplatePrinter.h>

Template parameter definition:

• Must start and end with a preset delimiter, the default is %
• Can only contain a-z, A-Z, 0-9, and _
• Maximum length of 63 characters (buffer is 64 including null).
• A parameter must not be zero length (not including delimiters).
• Spaces or any other character do not match as a parameter, and will be output as is.
• Valid examples
  • %MY_PARAM%
  • %SOME1%
• Invalid examples
  • %MY PARAM%
  • %SOME1 %
  • %UNFINISHED
  • %%

Template processing

A function or lambda is used to receive the parameter replacement.

bool templateHandler(Print &output, const char *param){
  //...
}

[](Print &output, const char *param){
  //...
}

Parameters:

• Print &output - the underlying Print, print the results of templating to this.
• const char *param - a string containing the current parameter.

The handler must return a bool.

• true: the parameter was handled, continue as normal.
• false: the input detected as a parameter is not, print literal.

See output in example 1 regarding the effects of returning true or false.

Template input handler

This is not needed unless using the static convenience function TemplatePrinter::start(). See example 3.

bool inputHandler(TemplatePrinter &printer){
  //...
}

[](TemplatePrinter &printer){
  //...
}

Parameters:

• TemplatePrinter &printer - The template engine, print your template text to this for processing.

Example 1 - Simple use with PsychicStreamResponse:

This example highlights its most basic usage.

//  Function to handle parameter requests.

bool templateHandler(Print &output, const char *param){

  if(strcmp(param, "FREE_HEAP") == 0){
    output.print((double)ESP.getFreeHeap() / 1024.0, 2);

  }else if(strcmp(param, "MIN_FREE_HEAP") == 0){
    output.print((double)ESP.getMinFreeHeap() / 1024.0, 2);

  }else if(strcmp(param, "MAX_ALLOC_HEAP") == 0){
    output.print((double)ESP.getMaxAllocHeap() / 1024.0, 2);
    
  }else if(strcmp(param, "HEAP_SIZE") == 0){
    output.print((double)ESP.getHeapSize() / 1024.0, 2);
  }else{
    return false;
  }
  output.print("Kb");
  return true;
}

//  Example serving a request
server.on("/template", [](PsychicRequest *request) {
  PsychicStreamResponse response(request, "text/plain");

  response.beginSend();
  
  TemplatePrinter printer(response, templateHandler);

  printer.println("My ESP has %FREE_HEAP% left. Its lifetime minimum heap is %MIN_FREE_HEAP%.");
  printer.println("The maximum allocation size is %MAX_ALLOC_HEAP%, and its total size is %HEAP_SIZE%.");
  printer.println("This is an unhandled parameter: %UNHANDLED_PARAM% and this is an invalid param %INVALID PARAM%.");
  printer.println("This line finished with %UNFIN");
  printer.flush();

  return response.endSend();
});   

The output for example looks like:

My ESP has 170.92Kb left. Its lifetime minimum heap is 169.83Kb.
The maximum allocation size is 107.99Kb, and its total size is 284.19Kb.
This is an unhandled parameter: %UNHANDLED_PARAM% and this is an invalid param %INVALID PARAM%.
This line finished with %UNFIN

Example 2 - Templating a file

server.on("/home", [](PsychicRequest *request) {
  PsychicStreamResponse response(request, "text/html");
  File file = SD.open("/www/index.html");

  response.beginSend();

  TemplatePrinter printer(response, templateHandler);

  printer.copyFrom(file);
  printer.flush();
  file.close();

  return response.endSend();
}); 

Example 3 - Using the TemplatePrinter::start method.

This static method allows an RAII approach, allowing you to template a stream, etc... without needing a flush(). The function call is laid out as:

TemplatePrinter::start(host_stream, template_handler, input_handler);

*these examples use the templateHandler function defined in example 1.

Serve a file like example 2

server.on("/home", [](PsychicRequest *request) {
  PsychicStreamResponse response(request, "text/html");
  File file = SD.open("/www/index.html");

  response.beginSend();
  TemplatePrinter::start(response, templateHandler, [&file](TemplatePrinter &printer){
    printer.copyFrom(file);
  });
  file.close();

  return response.endSend();
});

Template a string like example 1

server.on("/template2", [](PsychicRequest *request) {

  PsychicStreamResponse response(request, "text/plain");

  response.beginSend();

  TemplatePrinter::start(response, templateHandler, [](TemplatePrinter &printer){
    printer.println("My ESP has %FREE_HEAP% left. Its lifetime minimum heap is %MIN_FREE_HEAP%.");
    printer.println("The maximum allocation size is %MAX_ALLOC_HEAP%, and its total size is %HEAP_SIZE%.");
    printer.println("This is an unhandled parameter: %UNHANDLED_PARAM% and this is an invalid param %INVALID PARAM%.");
  });

  return response.endSend();
});

[zekageri]

Long ago, with AsyncTCP we discussed that the template character should be changed to a more complex type.
% is commonly used in CSS and in javascript strings. I think it would be much more mature to use a multi character template delimeter like EJS (https://ejs.co/) or moustache (https://github.com/janl/mustache.js) does it.

• In the case of EJS: <%= EJS %>
• In the case of moustache: {{mustache}}

[Chris--A]

Long ago, with AsyncTCP we discussed that the template character should be changed to a more complex type. % is commonly used in CSS and in javascript strings. I think it would be much more mature to use a multi character template delimeter like EJS (https://ejs.co/) or moustache (https://github.com/janl/mustache.js) does it.

• In the case of EJS: <%= EJS %>
• In the case of moustache: {{mustache}}

Good call, after I get my assignment submitted tonight I can adjust the code to allow variable length delimiters.

[Chris--A]

@zekageri
I'm doing a bit of an investigation, but to avoid unnecessary work, are there any compelling reasons to actually change from the %NAME% format. Sure a % may be used inside the code itself, but it doesn't really present an issue.

My implementation is different to others I've seen (AsyncWeb...), and example 1 shows that unhandled matches get printed out, so unless you are explicitly match a parameter and return true, the text is passed through as is anyway.

Additionally, the delimiter can be changed to any other character, so you could do #NAME#, $NAME$, ~NAME~.

I'm thinking it may be good to support a different prefix/suffix char so things like <NAME>, {NAME},[NAME], & (P_NAME) will work.

However, multi char prefixes and suffixes from what I can see will deliver little to no additional benefit.

[zekageri]

I can't recall what was the specific issue with async template but i will search for it asap. Maybe it was just broken from the start. I remember that if you had two classnames which both contained % char for a parameter the file broke. And there was a long discussion too. Will check!

[zekageri]

Here are the issues with one char templating

• Template variables' delimiter '%' conflicts with inline CSS me-no-dev/ESPAsyncWebServer#300
• Template confused by data containing percent ('%')  me-no-dev/ESPAsyncWebServer#333
• Using template processor corrupts response HTML if it includes % me-no-dev/ESPAsyncWebServer#644
• Template processor corrupts output me-no-dev/ESPAsyncWebServer#1234
• Poor selection of % as placeholder delimiter because is commonly used in CSS and JavaScript code. me-no-dev/ESPAsyncWebServer#1249

Some comments

• Template variables' delimiter '%' conflicts with inline CSS me-no-dev/ESPAsyncWebServer#300 (comment)
• Template variables' delimiter '%' conflicts with inline CSS me-no-dev/ESPAsyncWebServer#300 (comment)

[Chris--A]

@zekageri

Thank you very much for putting that together. Hopefully what I've written below will dispel some worries.

To start off with, sure the ESPAsyncWebServer version does have many issues, and is quite... something. This is the reason why I did a custom implementation.

However, my implementation does not have the issues posted above. It is a far simpler approach, and it strictly follows the rules outlined in the proposal.

me-no-dev/ESPAsyncWebServer#300

None of the strings surrounding % match a valid parameter, and the param handler is never called.

Here is an live example showing this: https://wokwi.com/projects/386442945617954817

me-no-dev/ESPAsyncWebServer#333

The answer to the issue below covers this, the output is not reprocessed.

me-no-dev/ESPAsyncWebServer#644
Just confirming: if we return "%" + var + "%";, we get stuck in an infinite loop where the template processor keeps trying to process the same thing over and over. It seems very determined to remove any % signs from the HTML.

As I write that, it becomes obvious to me that the example provided in the readme returns an empty String if var doesn't match the template string. So anything "surrounded" with % signs gets gobbled up.

There is a major difference between ESPAsyncWebServer and my handler functions:

ESPAsyncWebServer

String processor(const String& var)
{
  if(var == "HELLO_FROM_TEMPLATE")
    return F("Hello world!");
  return String();
}

TemplatePrinter

bool templateHandler(Print &output, const char *param){

  if(strcmp(param, "HELLO_FROM_TEMPLATE") == 0){
    output.print("Hello world!");
    return true;
  }
  return false; // Tell engine that input was not a parameter
}

1. ESPAsyncWebServer forces returning a string.
   TemplatePrinter allows you to say, no, that was not a parameter. In this case, the original matching string (with % delimeters) is output as is. Example 1 in my proposal shows this.

2. ESPAsyncWebServer reinjects the data into the processor.
   The output parameter of TemplatePrinter is the base output, no further processing is done. If you want to do a secondary parse, you can literally create a 2nd TemplatePrinter instance and run them through eachother before outputting to the base Print stream.

me-no-dev/ESPAsyncWebServer#1249 & me-no-dev/ESPAsyncWebServer#1234
2.) If you use % within a string or HTML file,
you must escape all occurrences of % - which are not enclose a placeholder - to %%.
Example: 'Width: 50%;' needs to be changed to 'width: 50%%;'

This is simply not needed with my implementation, only valid characters match a parameter.
I did find this though regarding %%:

This is now fixed, can view in the live example posted above:

asd%%asd: this will match an empty string. Currently everything will still work fine, the handler receives a string with null as the first character. However, I'll tweak the code to ignore empty parameters as there is little point triggering the handler. And as it doesn't match a valid parameter, it is technically a bug. Thank you for helping me find this.

And finally, with my template engine, the character is easily changed with the constructor:

TemplatePrinter printer(Serial, templateHandler, '~');

[Chris--A]

I have updated the code to ignore zero length parameters, not a harmful bug, but does not meet validity requirements.. Thank you @zekageri

[zekageri]

Well, it sounds like a whole lot better than Async's template parameter!
Thank you very much for your work, I will try it and hope @hoeken will merge it soon!

[hoeken]

Hey all, just wanted to say that I like what I see here and when you are ready to merge this in just let me know. I don't have any strong feelings on implementation, so I will leave it up to you @Chris--A

[Chris--A]

Yeah, lets do it. I was planning on allowing the size of the buffer (64 bytes) to be modifiable, but is a reasonable size. Not too short that complex parameter names can't be used, and not so long that it uses heaps of memory.

Certainly something I can look at if there is a compelling reason in the future.

[dzungpv]

@hoeken You should merge this, I am planing to replace ESPAsyncWebServer.

[hoeken]

Will do.

…

On Mon, Feb 26, 2024, 21:09 Pham Viet Dzung ***@***.***> wrote: @hoeken <https://github.com/hoeken> You should merge this, I am planing to replace ESPAsyncWebServer. — Reply to this email directly, view it on GitHub <#56 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABEHSUCKM74M6UBJB2OYM3YVU555AVCNFSM6AAAAABBPK2TBWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNRVGY3DCNJQGM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[hoeken]

Merged this as part of pull request by @dzungpv

Thanks again for your contribution.